BricksRL: A Platform for Democratizing Robotics and  Reinforcement Learning Research and Education with LEGO
Authors: Sebastian Dittert
Vincent Moens
Gianni De Fabritiis
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Robotics
Reinforcement learning
Primary area: robotics

Abstract: We present BricksRL, a platform designed to democratize access to robotics for reinforcement learning research and education. BricksRL facilitates the creation, design, and training of custom LEGO robots in the real world by interfacing them with the TorchRL library for reinforcement learning agents. The integration of TorchRL with the LEGO hubs, via Bluetooth bidirectional communication, enables state-of-the-art reinforcement learning training on GPUs for a wide variety of LEGO builds. This offers a flexible and cost-efficient approach for scaling and also provides a robust infrastructure for robot-environment-algorithm communication. We present various experiments across tasks and robot configurations, providing built plans and training results. Furthermore, we demonstrate that inexpensive LEGO robots can be trained end-to-end in the real world to achieve simple tasks, with training times typically under 120 minutes on a normal laptop. Moreover, we show how users can extend the capabilities, exemplified by the successful integration of non-LEGO sensors. By enhancing accessibility to both robotics and reinforcement learning, BricksRL establishes a strong foundation for democratized robotic learning in research and educational settings.

BricksRL: A Platform for Democratizing Roboties and Reinforcement Learning Research and Education with LEGO
Sebastian Dittert Universitat Pompeu Fabra sebastian. dittert@upf..ee edu
Vincent Moens PyTorch Team, Meta
vincentmoens@gaal.cc com
Gianni De Fabritiis
ICREA, Universitat Pompeu Fabra g ww.defabriiiss@@ee com
Abstract
We present BricksRL, a platform designed to democratize access to robotics for reinforcement learning research and education. BricksRL facilitates the creation, design, and training custom LEGO robots in the real world by interfacing them with the TorchRL library for reinforcement learning agents. The integration of TorchRL with the LEGO hubs, via Bluetooth bidirectional communication, enables state-of-the-art reinforcement learning training on GPUs for wide variety of LEGO builds. This offers flexible and cost-efficient approach for scaling and also provides robust infrastructure for conoo-eevrooomee-allrrrrri We present various experiments across tasks and robot configurations, providing built plans and training results. Furthermore, we demonstrate that inexpensive LEGO robots can be trained end-to-end in the real world to achieve simple tasks, with training times typically under 120 minutes on normal laptop. Moreover, we show how users can extend the capabilities, exemplified by the successful integration of non-LEEGO sensors. By enhancing accessibility to both robotics and reinforcement learning, BricksRI. establishes strong foundation for democratized robotic learning research and educational settings.
communication.
Introduction
As the field of artificial intelligence continues evolve, robotics emerges as fascinating area for deploying and evaluating machine learning algorithms in dynamic, real-life settings [14] 39 46).
These applications allow embodied agents to interact within complex environments, similar humans and animals, they must navigate variety of challenging constraints during their learning process.
Reinforcement learning (RL), in particular, has emerged promising approach learning complex behavior with robots 20 29 Despite the rich potential for innovation, the learning process of
algorithms under real-world conditions is challenge 38 46 The complexity of setting up robotics lab, combined with the high cost of equipment and the steep learning curve in RL, often limits the ability of researchers, educators, and hobbyists contribute to and benefit from cutting-edge developments. To address these challenges, we introduce BricksRL, comprehensive open-source framework designed democratize access robotics and RL. BricksRL builds upon Pybricks (44), a versatile Python package for controlling modular LEGO robotics hub, motors and sensors, actively maintained and supported by a vibrant community, and TorchRL (6). a modern framework for training
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

TorchRL Environment
TorchRL Agent
artion
Enviroinent
action
Robot
Rower Furdbe
Pybricks Class
Byle state
Clientpy
Tensortket rewaad (donz
Figure I: Communication overview of the agent, environment and robot.
RL agents. This synergy provides integrated solution that simplifies the creation, modularity, design, and training of custom robots the real world.
The use of LEGO parts as the basic building blocks for constructing the robots for BricksRL has the advantage of being cheap and widely available, which facilitates entry, but also makes easier to replace parts in the event of repairs and keeps costs down. In addition, the building blocks allow full reusability of all parts, which is not the case with other robots, and no special tools are required for construction or maintenance, which also keeps costs very low. By abstracting the complexities of robot programming given a gym-like like interface and RL algorithm implementation, BricksRL opens the door for a broader audience to engage with robotics research and education, making it more accessible researchers, educators, and hobbyists alike. The low cost, wide availability, and ease of deployment also allow the introduction and use of BricksRL as new artificial intelligence benchmark to test and evaluate RL algorithms robotics for a variety of tasks and robots.
The contributions of our work with BricksRL are threefold. First, we provide unified platform integrating Pybricks and TorchRL within BricksRL, which facilitates the physical control of cost- effective robotic systems and the design of gym-like training environments for RL algorithms. This setup enables scalable training across a diverse array of robots, tasks, and algorithms. Second, we demonstrate how to extend the capabilities of BricksRL beyond the standard sensor set provided by LEGO. By integrating camera sensor, we expand the platform's 's capabilities, allowing for the creation of more diverse robots and tasks and thereby broadening its potential applications. Third, we present preliminary results that underscore the framework's 's robustness and adaptability for real- world robotics applications. Furthermore, we provide explicit examples of sim2real transfer and the application datasets with offline RI., demonstrating the practical utility of BricksRL. in robotics and RL research. We make the source code available at: hhtpps: /ttps://gia htt///gtttubbu com/ C /r BarickkRL/bbl /briii/biiii cksrl The building plans, and evaluation videos of the experiments are publicly available at hhtps //brickerl. github........ io/ProjeccPaaee
2 Related Work
High acquisition costs, ranging from 10,000$ to over 200,000$, pose significant barriers to robotics research. This pricing affects various robotics types, including robotic arms (e.g., Franka [31],
Kuka [23) and advanced robotic hands 29 37 41 similar to costly quadruped or humanoid robots
designed for locomotion [5] 18 42
The popularization and increased consumer accessibility of 3D printing and DIY projects have heightened interest in low-cost robotics, thereby broadening access and facilitating the entry into roboties [1-3] 11 22, 32. lowering the initial costs for simple quadrupeds starting 300S robotic arms and hands for 20,000$. However, there is requisite need for access to 3D printer, workshop, and equipment for the construction, maintenance, and repair of these robots. Additionally, projects and companies have been established cater to the niche of low-cost robotics with pre-b robots for educational purposes that fall within a similar price range 4 35) Nevertheless, similar to off-the-shelf industrial robots, these and DIY robots are static, and it not assured that printed parts or other components can be repurposed for different robots or adapted for new tasks. This limitation often confines experiments to single robot and setting, which can be considered restrictive.
LEGO parts provide standardized and robust components that facilitate simple reconstruction, modu- larity of designs, and reproducibility. This modularity enables the construction and prototyping of various robots and the adaptation to different tasks, thereby simplifying the testing and benchmarking of RL algorithms across diverse robotic configurations. The initial cost for starter kit to construct

---

robots starts at approximately 400S [40), and can be augmented with additional sets, specific elements, or sensors as required. [36] demonstrates the application of LEGO for constructing robots for under 300$. However, using aluminum extrusions and 3D printed components, coupled with control via Raspherry Pi rather than LEGO's internal PrimeHub, diminishes the system's 's flexibility.
In contrast, the robots in BricksRL use only LEGO elements for construction and control. The simple integration of additional sensors is demonstrated but necessary. Further, users interact with the robots via a gym-like environment as is common in RL, simplifying the interaction. In comparison, the industry standard for managing robots and sensors the Robotics Operating System (ROS) [33] It offers numerous tools, however, its steep learning curve can be a barrier for researchers, students, hobbyists, and beginners starting with RL and robotics.
The use of LEGO for education robotics has a rich history through sets such as MINDSTORMS (66 or education sets [19]. These used not only in official educational institutions 25 26 43
but also in annual competitions around the world 11 45] that attract substantial number of children and students. To the best of our knowledge, these competitions do not currently incorporate machine learning techniques such as RL. Being tailored to these groups, BricksRL could bridge that gap and provide easy access to state-of-the-art algorithms.
BricksRL
The underlying architecture of BricksRL has three main components: the agent, the environment, and the robot[] TorchRL is utilized to develop the agent and the environment, while Pybricks is employed for programming on the robot side. In the following sections, we will examine each component individually and discuss the communication mechanisms between them.
3.1 Agents
BricksRL utilizes TorchRL's modularity to create RL agents. Modules such as replay buffers, objective functions, networks, and exploration modules from TorchRL are used as building blocks to create agents that enable a uniform and clear structure. For our experiments and to showcase the integration of RL algorithms within BricksRL, we have selected three state-of-the-a algorithms for continuous control: TD3, SAC, and DroQ [12] 15 [6]. We primarily chose these off-policy algorithms for their simplicity and their proven ability of sample-efficient training, which is essential as we mostly train our robots online. However, due to the flexible and modular structure of TorchRL, BricksRL can be easily adapted to include any algorithm developed within compatible with the TorchRL framework, allowing us to emphasize the general applicability of our system rather than the specific strategies. For example, methods commonly used in robotics, such imitation learning (47 and the use of foundation models (28). are available with TorchRI and can be seamlessly integrated into BricksRI.
3.2 Environment
BricksRL incorporates TorchRL's environment design principles among other components, to stan- dardize the structure and organization of its environments.
PybricksHub Class. Developed by BricksRL, the PybricksHub class plays an important role in facilitating communication with the LEGO Hub, which controls the robots. achieves this through Bluetooth Low Energy (BLE) technology, which enables efficient, low-power communication. This class is designed to manage a two-way data exchange protocol, critical for meeting the real-time requirements of interactive applications. Importantly, the PybricksHub class seamlessly bridges asynchronous communication with the traditionally synchronous structure of RI. environments.
EnyBase. In BricksRL, environments are designed as classes that inherit from EnvBase provided by TorchRL, which is foundational class for building RL environments. This structure gives access to the TorchRL ecosystem and simplifies the creation of environments. Users can create custom environments or extend existing environments for new tasks. All that needs to be done is to adapt the observation and action specifications and, if necessary, define a new reward function and adapt the step and reset function of the environment.
3

---

A key advantage of using TorchRL's EnvBase in BricksRL is the ability to apply environment transforms, fundamental feature of TorchRL. These transforms enable simple manipulation of the data exchanged between the environment and the agent. TorchRI. provides a wide range of useful transforms, such as frame stacking, observation normalization, and image transformations, which are particularly valuable for real-world robotics. Additionally, the integration of foundation models like VIP (28) through transforms expands the experimentation capabilities within BricksRL. Detailed descriptions of the environments implemented for our experiments, along with template for creating custom environments, can found A.2 and A.2.8 respectively.
Agent- Sgent-Enviromment Communication. For communication and data exchange between agent and environment, BricksRL. makes use TensorDict [6] as data carrier. TensorDict enables a standardized and adaptable transmission of information between agent and environment. TensorDict can handle a wide range of data as observation by accurately describing the observation specs in the environment, without modifying the agent's s or environment's 's essential structure. It enables users to shift between vector and picture observations or combination of the two. This is a crucial building component for being flexible to train different algorithms robots performing variety of tasks with and without sensors of the LEGO ecosystem.
3.3 LEGO Robots
In our experiments demonstrating the capabilities of BricksRL, we selected three distinct robot types to serve as an introductory platform for RL in robotics. These robots vary in complexity and their capacity for environmental interaction, reflecting progressive approach to robotic research. Addi- tionally, we incorporated various sensors and embraced a range of robot classifications, showcasing a broad spectrum of applications and use cases.
Figure 2: Three robots that we used in the experiments: (a) 2Wheeler, (b) Walker, (c) RoboArm.
2Wheeler. The 2Wheeler robot 2a built by us to represent an elementary robotic platform designed for introductory purposes, incorporating the LEGO's Hub, pair of direct current (DC) motors equipped with rotational sensors, and an ultrasonic sensor for determining the proximity to objects. The independent control capability of the DC motors endows the robot with high mancuverability.
Walker. The Walker 2b quadrupedal robot built in standard LEGO robotics kit, is equipped with four motors, each integrated with rotational sensors and an additional ultrasonic sensor. In comparison to the 2Wheeler robot 3.33 the Walker variant exhibits more degrees of freedom and a higher level of complexity due to its increased motor count and the fact that it uses legs instead of wheels. In terms of structural design, this robot bears similarity prevalent quadruped robots typically employed in the domain of locomotion control 39
The RoboArm 2c is built by and is similar to the Walker 4 motors with rotation sensors equipped, however, it has a higher range of motion. Further, it is the only static robot and includes another branch of robot types used for tasks like grasping and reaching manipulating objects 220 27]
RoboArm.
In general, Pybricks allows wide access to different motors and sensors as well as IMU (Inertial Measurement Unit) data in the LEGO's hub, which permits a variety possible modular robot archi- tectures and applications. For detailed overview, please refer to the official Pybricks documentation

---

30) Furthermore, we highlight the large community that has already created various robots, which can be rebuilt or used as inspiration. Any robot can be used with BricksRL as long as it is available in Pybricks's 8 interface. We welcome collaborations and encourage community members who want to experiment with BricksRI. to contact us for support.
Nobot-Environmentt Communication. BricksRL employs bidirectional data flow between the robot and the environment, facilitated by MicroPython in Pybricks. The agent's actions are transmitted as byte streams via standard input (stdin), where they are parsed and applied to the robot's motors. At the same time, the robot sensor data is sent back to the environment through standard output (stdout) for state evaluation and action selection. Each robot uses dedicated client script to manage its motors, sensors, and control flow for specific tasks. For exact details and an example of a typical client script, see the provided template in A.3
Communication Speed. In roboties and motion control, the rate of communication is crucial, necessitating high frequencies to ensure rapid responsiveness and precision in response to environ- mental changes or disturbances. Position or torque-based control systems in quadrupedal robots, for instance, operate within query frequency range of 20 to 200 Hz [8] 24). This frequency range enables these robots to swiftly adjust to variations in terrain during locomotion.
Likewise, the hub's control loop is capable of exceeding frequencies of 1000 Hz, making it suitable for managing complex robotic systems. Yet, when integrating with the BricksRL communication framework, reduction in the system-wide wide frequency, including agent-environment and environment- robot communications, introduced by utilizing stdin and stdout for communication. The process of reading from and writing to these streams, which requires system calls, inherently slower compared to direct memory operations. Additionally, the necessity to serialize and deserialize data through 'ustruct. unpack' and "ustruct.pack' adds to this overhead, as it requires converting data between binary formats used in communication and the Python object representation, which time-consuming.
Hz was observed. This decrease is primarily due to the overhead
Despite the overhead, BricksRL's communication speed, while on the lower spectrum, remains within reasonable range for robotic system applications. For instance, 13 39) have demonstrated that effective motion control in quadrupedal robots can be achieved at much lower frequencies, such as 20 or even 8 Hz, indicating that robust and dynamic locomotion can be maintained even at reduced communication frequencies. Moreover, we show our experiments that communication frequency is task and robot depending, for specific tasks optimal behaviors can be learnt faster with lower frequencies 7
3.4 Modularity and Reusability
The use of interlocking LEGO parts, and various sensors allows for endless possibilities in designing and building robots and robot systems. Additionally, precise construction plans and the reusability of components create additional opportunities. Unlike classic robot systems, users are not limited to single design and functionality. Instead, robots can be customized to specific requirements or tasks, and can even be reassembled for new challenges or ideas once the initial task is completed successfully. Building on this variable foundation with an infinite number of robotic applications, BricksRI. enables easy interaction by abstracting the complexities of the underlying communication processes. To train RL. algorithms, users can interact with the robots in gym-like environments, which provide a natural and intuitive interface. This enables researchers and hobbyists to train any RL algorithm, such as on-policy or off-policy, model-based or model-free.
To further illustrate modularity and scalability of BricksRL we extended the set of sensors and show how easy is integrate sensors outside of the LEGO ecosystem. Namely, we integrate a USB webcam camera into an environment (A.2.7) showeased in the experiments, demonstrating that additional sensors can further augment the scope of applications train robots with RL and BricksRI.
4 Experiments
In our experiments, we aim to address several critical questions regarding the feasibility and efficiency of training LEGO robots using RL algorithms in the real world with BricksRL. Thereby taking into
5

---

account the practical challenges of training LEGO robots such as the lack of millimeter precise robot constructions, the presence backlash, and noisy sensors.
Robot
Eavironments
2Wheeler
RunAway-
Spinning-vO
Walker
Walker-eD
WalkerSim- v0*
Roboarm
RoboArm-vO
RoboAmmin--vO"
SubeeAr--iixed-oe
Table 1: Overview of BricksRL Robot and Environment Settings. Environments marked with an asterisk (*) utilize LEGO sensors and image inputs as observations for the agent. Environments indicated by dagger (f) denote simulations of the real robot and do use the real robot for training.
Therefore we developed various task-specific environments to demonstrate the adaptability and case of use of BricksRL. highlighting the scalability of training across different algorithms and robots with diverse sensor arrays. Tasks ranging from driving and controlling the Wheeelr to learning to walk with the Walker and reaching tasks for the RoboArm demonstrate the applicability of BricksRL. Table shows complete overview of all environments used in our experiments.
In our experiments, we primarily focus on online learning, where the robot directly interacts with the real world, encompassing the challenges inherent to this approach. However, we have also developed simulation environments for certain tasks. Training in these simulations is significantly faster compared to real-world training, as confirmed by our comparative experiments. Additionally, we use these simulation environments to demonstrate the sim2real capabilities of LEGO robots with BricksRI.
A complete overview and description of the environments implemented including action and observa- tion specifications as well as the definition of the reward function can be found in the appendix [A.2] We also provide an environment template that demonstrates the straightforward process of creating custom environments using BricksRL.
In all of our experiments, initiated the training process with 10 episodes of random actions to populate the replay buffer. The results are obtained by over 5 seeds for each algorithm and compared against random policy. Evaluation scores of the trained policies
displayed in [ We further provide videos trained policies for each robot and task I Hyperparameter optimization was not conducted for any of the algorithms, and adhered default settings. Comprehensive information on the hyperparameter is provided the appendix 12 Although the option to utilize environment transformations, such frame stacking and action repetition, was available, opted not to use these features to maintain the simplicity of our setup. Further details are available in the appendix A.2
Environment
Algorithen
RunAway-v0
Spinning-vci
Walker-ao
WallkerSim-v0
RoboArm-vil
RoboArmSim--dl
RobuAm--issdd-V
2.31
7558.21
-12755
SAC
8.72
2.82
7407.20 20 109.50
52.04
8.79
55.00 4.95
-27.77
3.45
2.66
18.21
6.98
DroQ
8.96 + 8.77
7456-85 18:02
-67.63 63
10.44
-66.62
+ 2.81
55.02
62.45
-44004
26.05
-19.30
11.07
Random
0.51 + 1.84
71.97 + 501.79
191.90 00 + 18.19
191
+ 18.19
149.26 88
140.26 + 88.19
57.23
10.01
Table 2: The table displays the mean and standard deviation of evaluation rewards for the trained TD3, SAC, DroQ algorithms, and a random policy, based on experiments conducted across 5 evaluation episodes and different seeds.
4.1 2Wheeler
In the RunAway-vO task for the 2Wheeler robot, we trained RL algorithms over 40 episodes. Training sessions were completed in approximately 15 minutes per run for each agent. All algorithms successfully mastered the task, as shown in Figure 3 Notably, despite the simplicity of the task, algorithms adopted unique strategies. TD3 maximized its actions, achieving the highest distance from the wall but causing rapid acceleration, tilting, and noisy measurements, leading to occasional

---

abrupt episode termination. In contrast, the DroQ agent used smaller actions, resulting in more stable but shorter distances and avoiding premature episode endings A.5
For the Spinning-vO task, we trained the agents over 15 episodes. The training was completed in about 10 minutes, with all agents effectively learning solve the task, as illustrated in Figure Table includes the evaluation scores for both tasks of the 2Wheeler.
2Wheeler
Alporthmm::
SAC
Runvwwayy
Spinning-v0
10
8000
4000
Reserd
-2000
4000
-0000
-8000
Figure 3: Training results for 2Whecler robot for the RunAway-vO and the Spinning- environ- ment.
4.2 Walker
In the Walker-vo environment, we trained the Walker robot over 75 episodes. The entire training process took approximately 70 minutes. All agents successfully developed wrward-moving gait. Remarkably, the DroQ algorithm achieved this in significantly fewer episodes (5-10), requiring only about 15 minutes of training, as illustrated in Figure We trained the agents with a communication frequency of 2 Hz, instead of the maximum frequency of 11 Hz, as we observed that training at the lower frequency was faster and more stable. Results of a direct comparison can be found in the appendix 7
Figure also presents results from training in the WalkerSim-ve environment. To be comparable with the WalkerSim-vo, we similarly trained the agents for 75 episodes but noted marked differences training duration: TD3 and SAC completed within 1-3 minutes, whereas DroQ required about 20 minutes due to higher updating ratio. Simulation results revealed higher training performance with smoother and more stable learning curves compared real-world training
The final evaluation scores for policies trained in both real-world and simulation environments, tested on the actual robot, are summarized in Table 2 Although the simulation- iumulation-trained policies slightly underperformed, they demonstrated effective sim2real transfer, highlighting their efficiency with considerably less training time and reduced supervision.
Success Rate (%)
Algorithm
RoboArm-vo
RoboAm--mixd--v)
TD3
88
8
SAC
72
68
DroQ
64
68
Random
32
40
TD3*
SAC*
100
DroQ*
88
Table 3: Comparison of success rates for different agents in the RoboArm-vO and RoboArm-min -wi-mmiiiidoee environments. Success is defined as the agent reaching the goal or goal posi- tion within specified threshold. Agents marked with an asterisk (*) were initially trained in the RoboArmSim-vo environment. Each algorithm was evaluated for 5 epochs with 5 different seeds, totaling 25 experiments per agent and task.

---

Walker
Algnethmm:
SAC
D210 Wilkerslim.. uff
Willer
-50
150
-200
20
60
40
60
Episade
Figure 4: Training performance for Walker robot for the Walker-vO and the WalkerSim-v0 envi- ronment.
4.3 RoboArm
In the RoboArm-vO task, agents were trained for 250 episodes. Training durations varied, with TD3 and SAC completing in about hour the real robot, whereas DroQ required close to 2 hours. By contrast, training in the RoboArmSim- environment proved much quicker: TD3 and SAC finished within 1-2 minutes, and DroQ in approximately 25 minutes. The outcomes, depicted in Figure 5 confirm that all agents successfully learned effective policies.
To enhance the interpretation of the training outcomes, we also plotted the final error defined as deviation from the target angles at the last step each episode-and the total number of steps
taken per episode. The data reveals consistent decrease in both the final error and the number of steps throughout the training period. This indicates not only improved accuracy but also increased efficiency, episodes terminated sooner when goals were successfully met.
The evaluation results are detailed in Table Additionally, we compiled success rates that illustrate how often each agent reached the goal position within the predefined threshold. These success rates, derived from evaluation runs across 5 seeds with each seed running episodes, are presented Table 3] Notably, the policies trained in the RoboArmS ArmS ianSSmmvii v0 environment achieved superior evaluation scores and also higher success rates upon testing. This demonstrates successful sim2real transfer, achieving significantly reduced training time.
Lastly, present the training results for the RoboArm, mixed-v0 environment, where the algorithms underwent training over 250 episodes. Training durations varied significantly due to the complexity added by integrating additional image observations: SAC was completed in 40 minutes. TD3 in 60 minutes, and DroQ took three hours. The inclusion of image data likely introduced considerable noise the training results, as illustrated in Figure 6 which displays the rewards achieved by the agents and the corresponding episode steps. Interestingly, while SAC and DroQ successfully learned effective policies, TD3 struggled to adapt, failing to develop a viable strategy. The success of SAC and DroQ is evident in the chart of episode steps, showing a decrease in steps over the training period, which indicates a more efficient achievement the goal position.
The evaluation results, detailed Table 2 confirm the performances. Notably, out of 25 evaluation trials, both SAC and DroQ successfully reached the goal position 17 times, recorded in Table 3 This demonstrates the robustness of the SAC and DroQ algorithms in handling the complexities environment.
introduced in the
4.4 Offline Training
Offline RL uses pre-collected datasets to train algorithms efficiently, avoiding real-world interactions and complex simulations. To further highlight the capabilities of BricksRL. for education and research in robotics and RL we collected offline datasets for the LEGO robots. With those datasets, BricksRI. allows training of the LEGO robots via offline RI. imitation learning, both of which are Maate-ffthhe-ar methods for RL in robotics.

---

ReboAm
Algetth mm
Ranten
Doo
Fpiende Stops
RoboAm V0
250
-10.
150
50
100
100
Robokrm
200
150
50
-203
50
100
150
200
50
130
150
200
50 100
150
200
Episode
Figure 5: Training outcomes for the RoboArm robot in both the RoboArm-vO and RoboArmSim-vo environments. The plot also includes the final error at the epoch's 's last step and the total number of episode steps.
RoboAm www.mmxxeeoo--o
Algorihims:
Ransom
SAC
TD3
DVQ
-20
-40
Episode Steas
Revward
-80
120
Episoda
Figure 6: Training performance of the RoboArm robot in the RoboArm, RoboArm_mixed-vo environment, showing both the reward and the number of episode steps required to reach the target location.
For BricksRL, we curated datasets for three robot configurations: 2Wheeler, Walker, and RoboArm. These datasets include both expert and random data for four tasks in experiments (Walker-vO, Arm-vv, Awn-vvy-vO, Spinning-vO). Details about the datasets and dataset generation can be found in the appendix AA.7 Using these datasets, demonstrated that offline RL with BricksRL. is feasible, successfully training both online and offline RL algorithms and applying them to a real robot. The evaluation performance, shown in Table highlights the superior performance of offline RL algorithms, particularly with expert data, while online algorithms struggle, suggesting overfitting or poor generalization. For further details training parameters, please refer the appendix A.8
Walker-v0
RoboArm-v0 Expert
RunAway-vo
Spinning-v0 Expert
Agent
Random
Expert
Random
Random
Expert
Random
TD3
-79.71
-153 45
121.22
-201
9.06
6160.25
6168.28
SAC
66.91
-555.41
-54.67
-218.76
14.74
10.80
$416.11
9349,52
BC
-202.16 16
-85.65
117.772
-7.31
-0.27
18.13
35.55
9150.02
JOL 133..13
74.88
76.89
-3.10
14.07
18.50
4544.28
9U96.66
CQL 72.75
77..93
46.91
-17.41
19.60
19.74
4509.31
9099.03
Table 4: Evaluation Results: Online (TD3, SAC) and Offline (BC, IQL, CQL) RL Algorithms. Scores represent the mean reward averaged over 5 episodes and random seeds.

---

Conclusion
In this paper, we introduce BricksRL and detail its benefits for robotics, RL, and educational applications, emphasizing its cost-effectiveness, reusability, and accessibilityy. In addition, we showcased its practical utility by deploying three distinct robots, performing various tasks with range of sensors, across more than 100 experiments. Our results underscore the viability of integrating state-of-the-art RL methodologies through BricksRL within research and educational contexts. By providing comprehensive building plans and facilitating access to BricksRL, we aim to establish this investigation as foundational proof of concept for utilizing LEGO-based robots to train RL algorithms. Moving forward, avenues for further research include creating more complex robots and tasks, exploring applications in agget settings, and leveraging large datasets to enhance RL training through terssfommr-bbsct imitation learning. Ultimately, BricksRL sets the stage for a future where accessible, reusable robotic systems support and expand RL research, collaborative learning, and interactive education.
Acknowledgements
We thank A. De Fabritiis Campos, M. De Fabritiis Campos and P. Vallecillos Cusco for providing their LEGOs to this project.
References
[1] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. "ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots". en.
In: Proceedings of the Conference Robot Learning. ISSN: 2640-3498. PMLR, May 2020,
pp. 1300-1313. URL: httph:://////t http::///procceeinng mlr press v1000 04/09/2024).
html (visited
[2] Jorge Aldaco, Travis Armstrong, Robert Baruch, Jeff Bingham, Sanky Chan, Debidatta Dwibedi, Chelsea Finn, Pete Florence, Spencer Goodrich, Wayne Gramlich, Alexander Herzog, Jonathan Hoech, Thinh Nguyen, Ian Storz, Baruch Tabanpour, Jonathan Tompson, Ayzaan Wahid, Ted Wahrburg, Sichun Xu, Sergey Yaroshenko, and Tony Z Zhao. "ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperation". en. In: O.
[3] Raña Bindu, Sazid Alam, and Asif Neloy. "A Cost-Efficient Multipurpose Service Robot using Raspberry Pi and 6 DOF Robotic Arm". In: Mar. 2019, pp. 16 12-22. DO1: 10. 1345/332569
3325701
[4] Rinu Boney, Jussi Sainio, Mikko Kaivola. Amo Solin, and Juho Kannala. RealAnt: An Open- Source Low-Cost Quadruped for Education and Research in Real-World Reinforcement Learn-
ing. arXiv:201 1 003085 [cs]. June 2022. DOT: 10 48550/ arxiv. 2011. 03085
URL: http
// //arxiv..or/ wttpiii.rrggbb///// 2011 2011 03085 (visited on 04/09/2024).
[5]
Boston Dynamics' Spor Robot Dog Now Available for $74,500 JEEE Spectrum. en. URL: https: https://sspectrttt ieee.
(visited on 05/01/2024).
[6] Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. TorchRL: A data-driven decisign-making library for PyTorch. arXiv:2306.0051 [cs]. Nov. 2023. DOI: 10. 485448855050 48550/arXiv arxiv..iivvv.v... 2306. 00577 00577 (visited on 05/01/2024).
URL: http:///rxii. org.
[7] Ken Caluwacrts, Atil Iscen, J. Chase Kew, Wenhao Yu, Tingnan Zhang, Daniel Freeman, Kuang-Huei Lee, Lisa Lee, Stefano Saliceti, Vincent Zhuang, Nathan Batchelor, Steven Bohez, Federico Casarini, Jose Enrique Chen, Omar Cortes, Erwin Coumans, Adil Dostmohamed, Gabriel Dulac-Arnold, Alejandro Escontrela, Erik Frey, Roland Hafner, Deepali Jain, Bauyrjan Jyenis, Yuheng Kuang, Edward Lee, Linda Luu, Ofrr Nachum, Ken Oslund, Jason Powell, Diego Reyes, Francesco Romano, Feresteh Sadeghi, Ron Sloat, Baruch Tabanpour, Daniel Zheng, Michael Neunert, Raia Hadsell, Nicolas Heess, Francesco Nori, Jeff Seto, Carolina Parada, Vikas Sindhwani, Vincent Vanhoucke, and Jie Tan. Barkour: Benchmarking Animal- level Agility with Quadruped Robots. en. arXiv:2305. 14654 [cs]. May 2023, URL: http //aaxiv.org//// 7/axxiv. www./rrgg//.......ooooooooo..../ aba/abb//5.... 14654 (visited on 05/1 05/13/2024).
10

---

[8] Shuxiao Chen, Bike Zhang. Mark W. Mueller, Akshara Rai, and Koushil Sreenath. Learning Torque Control for Quadrupedal Locomotion. arXiv:2203.05 0212200305194 [es, eess]. Mar. 2023. URL: http: /ttt:///axiv. www. abs/ /20.. 05194 (visited on 03/06/2024).
[9] Mare Peter Deisenroth and Carl Edward Rasmussen. "PILCO: A Model-Based and Data- Efficient Approach to Policy Search". en. In: O. 8.
[10] Hongjie Fang, Hao-Shu Fang, Yiming Wang, Jieji Ren, Jingjing Chen, Ruo Zhang, Weiming Wang. and Cewu Lu. Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild. arXiv:2309. 14975 [cs]. Sept. 2023. URL: http: / /wrxiv org/ab// 2309. 14975 (visited on 04/22/2024).
[11] Zipeng Fu, Tony Zhao, and Chelsea Finn. "Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Telcoperation"
In: O.
[12] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. Number: Xiv: 880.09977 arXiv: 1802.09477 [cs, stat]. Oct. 2018.
URL: http:tttp//xiii.... org/ abs 1802. 09477 (visited 06/20/2022).
[13] Siddhant Gangapurwala, Luigi Campanaro, and Ioannis Havoutis. Learning Low-Erequency
Motion Control for Robust and Dynamic Robot Locomotion. arXiv:2209, 14887 [cs]. Feb. 2023. DOI: 10. 48550/ arXiv 2209. 14887 (visited on 03/06/2024).
URL: HRL::htt::/// httppp//aaaxiv.orgaabb////////// wtgg/ba//gaaaaaaa/aabb//////// 14887
[14] Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H. Huang, Dhruva Tirumala, Markus Wulfmeier, Jan Humplik, Saran Tunyasuvunakool, Noah Y. Siegel, Roland Hafner, Michael Bloesch, Kristian Hartikainen, Arunkumar Byravan, Leonard Hasenclever, Yuval Tassa, Fereshtch Sadeghi, Nathan Batchelor, Federico Casarini, Stefano Saliceti, Charles Game, Neil Sreendra, Kushal Patel, Marlon Gwira, Andrea Huber, Nicole Hurley, Francesco Nori, Raia Hadsell, and Nicolas Heess. Learning Agile Soccer Skills for Bipedal Robot with Deep Reinforcement Learning. arXiv:2304. 13653 [cs]. Apr. 2023. URL: http: //tp///aavvvv/// org/ abs/ 2304. 13653 (visited on 03/04/2024).
[15] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor- Critic Algorithms and Applications. arXiv: 1881.05990 [cs, stat]. Jan. 2019. URL: http
//arxiv.ors at////ass////// 05905 (visited on 03/06/2024).
[16] Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout Q-Functions for Doubly Efficient Reinforcement Learning. arXiv:21 10.00044 [cs]. Mar. 2022. URL: http:://axii: org/ hetg//b/221000 2ttp//23100000444411111 (visited on 03/16/2024).
[17] Home Page 1 FIRST LEGO League. en. URL: https: (visited on 04/30/2024).
www firstlegoleague org/
[18] Marco Hutter, Christian Gehring, Dominic Jud, Andreas Lauber, C. Dario Bellicoso, Vassilios Tsounis, Jemin Hwangbo, Karen Bodie, Peter Fankhauser, Michael Bloesch, Remo Diethelm, Samuel Bachmann, Amir Melzer, and Mark Hoepflinger. "ANYmal a highly mobile and dynamic quadrupedal robot". In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). ISSN: 2153-0866. Oct. 2016, pp. 38-44. DOL: 10. 1 1009/IROS URL: httts:://teeee/eee ieee. org orgg//ocumet//755809 (visited on
2016. 7758092 05/01/2024).
[19] Juguetes de LEGO@ Education Oficial LEGO@ Shop ES. es. URL: https: wwwp/w..... lego hean//s-es//themese wwtp:/lleoooooeeeeeeeeeeeeeeeeee education. (visited on 05/13/2024).
[20] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv:1806. 10293 [cs, stat]. Nov. 2018. DO1: 10 48550/ 18550/arxiv... arxiv. 1806. 10293 10293 (visited on 03/04/2024).
URL: http
// //axxiv.orpp http:/ooggbbbbbbbboeo
[21] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. MT-Optt: Continuous Multi Robotic Reinforcement Learning at Scale. arXiv:2104.082 [cs]. Apr. 2021. DOI: 10. 48550/85000 arrivv 2104. 08212 URL:
http://////aaaaa///////////////. 08212 (visited on 03/04/2024).
[22] Nathan Kau. Stanford Pupper: A Low-Cost Agile Quadruped Robot for Benchmarking and Education. en. arXiv:21 10.00736 [cs]. Feb. 2022. 00736 (visited on 04/30/2024).
URL.: http: arxiv org/ abs absaaba//111 /210

---

[23]
LBR iiwa. DE URL: httph: kuka. com httpa////oo/.///ppooooo//oooo/// industrrall-rbbos/lt--: (vistti on 05/01/2024).
[24]
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. "Learn-
ing Quadrupedal Locomotion over Challenging Terrain". In: Science Robotics 5.47 (Oct. 2020).
arXiv:2010.1 1251 [cs, eess], eabc5986. ISSN: 2470-9476 DOI: 10. 1126 scirobotics
abc5986 URL: http://aaxwv..rg///aa.a/. 2010 11251 (visited on 03/06/2024).
[25] Lego Robotics Wagner College Lifelong Learning. en-US, URL: Stttps:/wwagnn edu/
(visited on 05/13/2024).
[26] Lego Robotics Competiliion Fort Hays Stute University (FHSU). en. URL: httph:ttp/////.
hatu wodusseei/eeoooooeeee wwwotoicooaoooooo robstica/(visieed
(visited on 05/13/2024).
[27] Sergey Levine, Peter Pastor, Alex Krizhevsky. and Deirdre Quillen. Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection.
(IXiv:1033022199 [cs]. Aug. 2016. DOI: 10 48550 / arxiv. 1603. 02199 URL: http
//arxiv. abw//603........../... 02199 (visited 03/04/2024).
[28] Jason Yecheng Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. "VIP: TOWARDS UNIVERSAL VISUAL REWARD AND REPRESENTATION VIA VALUE-IMPLICIT PRE- TRAINING". en. In: O.
[29] OpenAI, lige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving Rubik's Cube with Robot Hand. 1911v01910007113 [cs, stat]. Oct. 2019. DOI: 18.48550/ /tt///aiivv./..///// (visited on 03/04/2024).
URL: http:///aaaaiiioorra////9/......
[30] Pybricks Documentation - pybricks v3.5.0 documentation. URI. httpp:://bcssppyriic..
com/ ee stt////beeee/iiiiieedde/
(visited on 04/30/2024).
[31]
Research. en. URL: https:tttttttttiee http://aanaaadde/sssrraaall (visited on 05/01/2024).
[32]
Paulo Rezeck, Hector Azpurua, Mauricio FS Correa, and Luiz Chaimowiez. HeRo 2.0:
Low-Cost Robot for Swarm Robotics Research. arXiv:2202.12391 [cs]. Mar. 2022. DOI:
10. N0.48550/aa 103550/arii..o 03/04/2024).
arXiv. 2202. 12391
URL: htl::http:/// btttp://arxivv org/ http://.a/22222122222 12391 (visited on
[33] Robot Kits & STEM Toys K-12 Schools and Home EduccationlMakeebiooa en. URL: https //www www.blckee....... com/ (visited on 05/13/2024).
[34] Robotik in Schulen - die Robotik für Bildung 4.0. de-DE. URL: hotp::tttte:/rrrroooiic.oo
produkk-kategoriinen wttp--aaterrrretoooaeeoe------ee schulen/ (visited on 05/13/2024)
[35] Robotis The study of humanexperiiencss. en. URL: htttps://eet htt.//iirroooootocoooo (visited on 05/13/2024).
covisiitee
[36] Logan Saar, Haotong Liang, Alex Wang. Austin McDannald, Efrain Rodriguez, and A Gi- lad Kusne. "A Low-Cost Robot Science Kit for Education with Symbolic Regression for Hypothesis Discovery and Validation". en. In:
[37] Shadow Dexterous Hand Series Research and Development Tool. en. Running Time: 66. (visited
Sept. 2023. URL: https: //tpps://www Shadorrrboo..com/aatttrooooo on 05/01/2024).
[38] Laura Smith, Yunhao Cao, and Sergey Levine. Grow Your Limits: Continuous Improvement with Real- World RL for Robotic Locomotion. arXiv:2310. [cs]. Oct. 2023. Do1: 10 48550/ / arXiv 2310 17634 URL: http: 03/04/2024).
arxiv / org/ / abs 2310 17634 (visited on
[39] Laura Smith, 1lya Kostrikov, and Sergey Levine. Walk in the Park: Learning Walk in 20 Minutes With Model-Free Reinforcement Learning. arXiv:2208 [cs]. Aug. 2022. DOI: 10. 4.548/50//5 arXiv. 2208 07860 03/04/2024).
URL: http: http::///axxvieiee org/ wwwp:/wwwwwww/////////////////// awtpaaa////2/ 07860 (visited on
[40] STEM & STEAM Solutions for the Classroom. en-us. URL: https http:://duccaooonncc lego
com/ en cen-us/ (visited on 05/13/2024).
[41]
The Dexterous Hands INSPIRE. en-US. URL: https: // /n inppire-robots. com dexterous hands (visited on 05/01/2024).
Product-categorryy hiateeoory/thee-dxxeertuouu
[42]
UNITREE Robotics@ SHOP Official Unitree Robot Dogs Al Gol. en-US. URL: https
/umiirrerroboics..ou net/ (visited on 05/01/2024).
12

---

[43] Carnegie Mellon University. LEGO Curriculmm Carnegie Mellon Robotics Academy
Carnegie Mellon University. en. URL: https / www cmu edu wtwpooooooccceaee....e
htm (Visited on 05/13/2024).
[44]
Laurens Valk. Pybricks. URL:
(visited on 04/30/2024).
[45]
WRO RoboMission LEGO Roboter lösen Aufgaben auf WRO Parcours. URL: https: /
www #orldrobotolyypiaal de /eoewooldd robot
(visited on
05/09/2024).
[46] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Day- Dreamer: World Models for Physical Robot Learning. en. arXiv:2206. 14176 [cs]. June 2022.
URL: http://arxiv org/ abs/ 2206. 14176 (visited 04/26/2024).
[47] Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges. aw.iv:230i 022473 [cs, stat]. Sept. 2023. DOI: 10. 485448550//0 arXiv. 2309. 02473 (visited on 05/22/2024).
URL: http::tttt:///tt/ org.abgg//....... abs///b///// 2309. 02473
A Appendix
A.1 Repository and Website
BricksRI. repository and our project website with evaluation videos and building instructions can be found at the following locations:
GitHub: BricksRL
Project Website
A.2
Environments
A.2.1
RunAway-vo
The RunAway-ve vo environment presents a straightforward task designed for the 2Wheeler robot. The objective is maximize the distance measured by the robot's ultrasonic sensor. To accomplish this, the agent controls the motor angles, deciding how far to move forward or backward. This environment operates within continuous action space, and each episode spans maximum of 20 steps. Episodes will terminate early if the robot reaches the maximum distance of 2000 mm.
Action and Observation Specifications. TorchRL's BoundedTensorSpecs the action and observation specifications, which have and dimensions, respectively. Table outlines the specific ranges. Actions initially defined in the range of [-1, 1] but are linearly mapped to the range of 100, 100] before being applied to both the left and right wheel motors.
employed to define
Type
Num
Specification
Min
Max
Action Spec
0
motor
-1
1
0
left motor angle
0.0
360.0
right motor angle
0.0
360.0
Observation Spec
2
pitch angle
-90
90
3
roll angle
-90
90
4
distance
0.0
2000.0
Table 5: Combined action and observation specifications for the RunAway-vO environment.
Reward Function.
The reward function for the RunAway-vO environment defined as:
+1 if distance, > distance,-1
if distance, < distance;-1
(I)
else
13

---

A.2.2 Spinning-v0
The Spinning-vO environment is another setup designed for the 2Wheeler robot. Unlike RunAway-vO, this environment does not use the ultrasonic sensor measure the robot's distance to objects in front of it. Instead, at each reset, value is randomly selected from the discrete set 0, 1, which explicitly dictates the rotational direction which the robot should spin. The robot's IMU (Inertial Measurement Unit) enables the tracking of various parameters, including angular velocity. This angular velocity is part of the agent's observation the Spinning-ve environment, providing with information its rotational direction to complete the task. The action space is also continuous, and each episode has a length of 50 steps.
Action and Observation Specifications. The BoundedTensoorSpe for the observation in the Spinning-Ve v0 environment comprises six floating-pon
values: the left and right motor angles, the pitch and roll angles, the angular velocity wiz and the rotational direction. Table 6 details these specifications. The action specification is defined as two floating-poit values representing the rotation angles applied to the left and right motors. Actions are initially defined in the range of [-1, 1] but will be transformed to the range of 100, 100] before being applied to the motors, as detailed in Table
Type
Num
Specification
Min
Max
0
left motor
-1
Action Spec
right motor
-1
0
left motor angle
0.0
360.0
right motor angle
0.0
360.0
2
pitch angle
-90
90
Observation Spec
3
roll angle
-90
90
4
angular velocity wa
-100
100
5
direction
0
Table 6: Combined action and observation specifications for the Spinning- vo environment.
Reward Function. angular velocity wa is directly used as a reward signal and encourages adherence to a predefined rotational orientation.
The reward function for the Spinning-vi vO environment is defined in [ The
Weat
if rotational direction = 0 (spinning left), otherwise (spinning right).
Wa.t
A.2.3 Walker-v0
In the Walker-vO environment for the Walker robot, the objective is to master forward movement using its four legs. To achieve this, the robot is provided with data on the current angles of each leg's 's motors, along with IMU readings that include both pitch and roll angles, ensuring operational safety. Additionally, an ultrasonic sensor is used to momentarily halt the agent's actions when the detected distance falls below a predefined threshold, preventing collisions with obstacles. Each episode consists of 100 steps. To achieve reduced communication speed, waiting time added after the actions are applied to the motors.
Action and Observation Specifications. In the Walker-vo environment, the observation specifi- cation consists of seven floating-poin values: four motor angles (one for each leg), the pitch and roll angles, and the distance measurements from the ultrasonic sensor. The action specification includes four floating-poin point values corresponding to the four leg motors. These action values are initially defined in the range of [-1, 1] but are linearly mapped to the range of [-100, 0] before being applied, as detailed in Table 7
Reward Function. The total reward is a sum of penalties for the actions and differences in angles, encouraging synchronized movement and appropriate angular differences between the legs of the
14

---

Type
Num
Specification
Min
Max
left front motor
-1
right front motor
-1
Action Spec
2
left back motor
-1
3
right back motor
-1
0
left front motor angle
0.0
360.0
right front motor angle
0.0
360.0
2
left back motor angle
0.0
360.0
Observation Spec
3
right back motor angle
0.0
360.0
pitch angle
-90
5
roll angle
-90
90
distance
0.0
2000.0
Table 7: Combined action and observation specifications for the Walker-vO environment.
walker. The reward components are defined as follows: : Penalty for the magnitude of actions taken. ri Penalty for the angular difference between the left front (If) and right back (rb) motor angles. Re-ub, Penalty for the angular difference between the right front (rf) and left back (Ib) motor angles. Ru-st Penalty for the deviation from 180 degrees between the left front (If) and right front (rf) motor angles. RIb-bbr: Penalty for the deviation from 180 degrees between the left back (Ib) and right back (rb) motor angles.
R, Raction, Ri + Ret + Ratat, + Rib
(3)
Σ actions;
Raction,
(4)
40
Rigbbt angular difference(fir.. 0z..1)
(5)
180
angular_diin differencee 01b,t)
(6)
180
REseft 180 - a-gglangggdiiiiii differencee(fr.. Oft,e)
(7)
180
Ribab,t 180 - angular diffcrence(On... Oxb.e)
(8)
180
with the angular difference defined as:
1((02, - 01,e 180) mod 360) 180
(9)
A.2.4 WalkerSim-v0
Additionally, we have developed a simulated version of the Walker-v0 environment, called WalkerSim-vO. This simulation mirrors the real-world setup without requiring communication with the actual robot or using PyBricks. the simulation, both IMU measurements and ultrasonic sensor inputs, which are used in the Walker-vo environment, are set to zero. This simplification made because modeling or simulating these sensors can be challenging due to their complexity and the nuances involved in accurately replicating their readings. However, since these sensors primarily used for safety in the real world, their absence is not concern in the simulated environment.
In WalkerSim-vo, the next motor states are calculated by simulating the transition dynamics. This involves transforming the action output into the angle range and then applying the corresponding actions by adding to the current motor state. To model real-world inaccuracies, we add noise to the motor states, sampled from Gaussian distribution with mean of 0 and a standard deviation of 0.1. Similar to the real-world environment the WalkerSim-vo episodes consist of 100 interactions.
15

---

Action and Observation Specifications. the Walker-vOB
Action and observation specifications are the same as in
Reward Function. environment.
The reward function for the WalkerSim-vO the same as in the Walker-vO
A.2.5 RoboArm-v0
The RoboArm-vO is pose-reaching task. At every reset, random goal angles for the four motors are sampled, defining target pose. The objective is adjust the robot's articulation to reach the specified goal pose within 100 steps. To achieve this the robot is provided with the current motor angles of all joints. The design of this environment permits the user to choose whether the robot tackles the task with dense or sparse rewards, effectively adjusting the difficulty level.
Action and Observation Specifications. The observation specification for the RoboArm-vo en- vironment consists of eight floating-poing values: four current motor angles and four goal motor angles, as detailed in Table 8 The action specification is defined by four floating-poit poating--pint values for the four motors, initially in the range of [-1, 1]. Before applying the specific actions each motor, these values are transformed as follows: the rotation motor actions are linearly mapped to the range [-100, 100], the low motor actions [-30, 30), the high motor actions to [-60, 60), and the grab motor actions to 2-2 25].
Type
Num
Specification
Min
Max
0
rotation motor
-1
low motor
Action Spec
2
high motor
-1
1
grab motor
rotation motor angle
0.0
360.0
low motor angle
10
70
high motor angle
-150
10
grab motor angle
-148
-45
Observation Spec
goal rotation motor angle
0
360
goal low motor angle
10
70
goal high motor angle
-150
10
goal grab motor angle
-148
-45
Table 8: Combined action and observation specifications for the RoboArm-vO environment.
Reward Function. The reward function for the RoboArm-vO environment can be chosen to be either dense or sparse. In the sparse case, the reward if the distance between the current motor angles and the goal motor angles is below defined threshold; otherwise, it is O. In our experiments, however, we used the dense reward function, which calculated follows:
| deg
R2
(10)
100
where Afdeg is the vector of shortest angular distances time step t between the goal motor angles gooat and the current motor angles Ouument,t, defined as:
Adgg degrees aretan (sin(raiians(@goal. radians (Feurrent,e)
(1I)
cossraiians@@gon... radians (Quuncentt)))
16

---

A.2.6 RoboArmSim-v0
RoboArmSim-vo mirrors the real-world RoboArm-vO environment but is entirely simulated, removing the need for physical interaction with an actual robot or the use of PyBricks. In this virtual setup, the RoboArm's 's task remains pose-reaching challenge, where it must align its articulation match randomly sampled goal angles for its four motors, setting a target pose at every reset. The robot in the simulation is provided with the current motor angles all joints and the goal angles accomplish this task within 100 steps. Crucially, the simulation is straightforward since it does not require modeling or simulating complex sensor measurements for state transitions, but only the motor angles, simplifying the simulation of state transitions significantly. Similar to the WalkerSim-we we add Gaussian noise (0, 0.05) to the actions before the linear mapping and addition with the current motor states. Like its real-world counterpart, this environment allows users to choose between dense and sparse reward structures, facilitating the adjustment of the task's difficulty level.
Action and Observation Specifications. Action and observation specifications are the same as in the RoboArm-vO
Reward Function. RoboArmSin-vo environment we use the same dense reward function as defined for the RoboArm-vO environment.
A.2.7 RoboArm-mixxd-vy
In the RoboArm_mixed-vO environment, an additional sensor input is available to the robot through webcam. The RoboArm holds red ball in its hand and must move it target position. Each episode has maximum of 30 steps. The target position is randomly selected and displayed as green circle in the image. The image determine if the conditions to solve the task are met.
additional information for the algorithm and used to
Action and Observation Specifications. The full observation specifications for the RoboArm_ filarmmmixed_ mixed-vo environment consist of three floating-pun teating-poiittoee values representing the three motor angles (rotation motor, low motor, high motor) and image observation specifications with a shape of (64, 64), as detailed in Table
The action specifications are also three floating-poin values in the range of [-1, 1], which will be transformed before being applied to the specific motor. The rotation motor angles are transformed to the range of |-0 90), the low motor angles to the range of [-30, 30), and the high motor angles to range of [-60, 60).
Type
Num
Specification
Min
Max
0
rotation motor
-1
Action Spec
1
low motor
-1
2
high motor
-1
0
rotation motor angle
0.0
360.0
Observation Spec
1
low motor angle
10
2
high motor angle
-150
10
Image Observation Spec
O
image observation Size: (64, 64)
0
255
Table Combined action and observation specifications for the loboArm_mixed-e environment.
Reward Function. To calculate the reward for the mixed observation environment bebooArm_mixed--ee we utilize the Python package OpenCV to detect the red ball and measure the distance to the target location depicted in the image. First, we convert the image from BGR to HSV and define a color range to identify the contours. For each detected contour, we calculate the distance to the center of the green target circle and take the mean distance the reward:
17

---

Σ distances:
R₂
(12)
n4 100
where is the number of distances detected at the current time step. If no contours are detected, we use the previous reward as the current reward.
A.2.8 Task Environment Template
To illustrate the simplicity of using the TorchBricksRL BaseEnv, which manages communication
between the environment and the robot, we provide an example Listing [ template for creating new custom environments.
This code can serve as
import torch
from environments. base. base_ env import BaseEnv from tensordict import TensorDict, TemsorDictBase
from torchrl. data. sensor_specs import BoundedTensorSpec, CompositeSpec
class TaskEnvironment (BaseEnv):
Define your action and state dimension, needs to be adapted depending on your task and robot! action_dim
# One action control the wheel motors together # sensors readings (left motor angle, right motor angle, pitch, roll, distance)
11
state_dim
12
Define observation space ranges.
motor_angles [0. 360]
15
roll_angles [-90, 90]
16
pitch_angles [-90, 90]
17
distance [o, 2000]
19
observation_key "vec_observation"
20
21
def _niit__ self,
):
24
self baff.bbatch_siee torch. Size ([1])
25
26
# Define Action Spec.
self action_
BoundedTensorSpec (1ow=-1, high=1, shape= (1,
self. action_dim))
28
# Define Observation Spec.
29
bounds = torch.
self motto_angless
32
self mow.r angles,
33
self rolfrrollangles
34
self www.ppichhheeeeeee
self distance,
]
37
38
observation_spe spec = BoundedTensorSpec (
low=bounds [;, 0]
nigh-bounds [:, 1],
42
self observation.speeeeee.... CompositeSpec ((self observation..kyy
Dbservation_spec? shape=(1,))
super init__ action action_dim, state_ wtt.sss....ttte....
45
46
47
18

---

def _reset (self, tensordict: TensorDictBase, **kwargs) -> TensorDictBase:
50
# Get initial state from hub.
51
observation self read.eee__rromubb
82
# Could also add external sensors here and return them as well
# img self read return TemsorDict(
54
55
56
self observation_key norm_observation. float O) image_obs: img float (),
58
59
batch_ size__,,
60
def (self, state, action, next_state) -> Tuple(float, booll: # Define your reward function.
63
64
65
reward, done 0, False return reward, done
67
68
def _step tensordict: TensorDictBase) -> TensorDictBase:
69
# Send action to hub to receive next state.
action = petoorddcctgetaaatiiioi self teute (action)
cp numpy () squeeze (0)
12
# Read next state next_observataee = self fead_from_hub
hub.
23
74
# Compute the reward. state tensordict gestt.getteellll wtw.giaaal.ee..oosrraatioo...e.. next_state next_ textrricn get (self.
76
77
78
reward, done reward state-state,
80
action=action, noxt_state-next.set
81
82
# Create output TensorDict next tensordict TensorDict(
85
86
self observation__kyy self normalize_stata next_observation fllaaitntt............ f),
"reward": torch. tensor ([revard]) float O)
"done": torch. tensor [done]) (),
89
} siteh_
90
device=ttesorddict device,
93
return next tensordict
Listing 1: Task environment template
A.3
Client Script
For each task and robot, a custom client script is required facilitate interaction between the robot and the environment. The client.py script defines the configuration of motors, sensors, and the workflow for processing and exchanging data. This script must be uploaded to the Pybricks Hub and updated whenever the robot's configuration changes, such as when motors or sensors are added or removed. Listing 2 provides a simple example of client script tailored for the RunAway-vO task. In this example, single float value representing the action is used to control the motors, while sensor data collected and transmitted back to the environment.
import ustruct
from micropython import intr
19

---

from pybricks hubs import InventorHub
from pybricks parameters import Direction, Port
from pybricks pupdevices import Motor, UltrasonicSensor import DriveBase from pybricks. tools import wait
from pybricks.
from uselect import poll
from uays import stdin, atdout 10
11
12 Initialize the Inventor Bub.
hub InventorHub()
14
18 Initialize the drive base.
16 left_ Motor Motor(Port.. (Port. Direction. D COUNTERCLOCKWISE)
right__otor Motor (Port
drsve_base DriveBase (left_motor, right_motor)
19 Initialize the distance sensor.
20
sensor UltrasomicSensor (Poree (Port.C)
keyboard - p)ll()
23
keyboard... register (stdin)
24
25
while True:
Optional: Check available input. while not keyboard poll (0):
28
29
w1)
30
Read action values for the motors. action ustruct. sapfck("!f,, stdin. buffer.ufffr... read (4)) [o] Apply the action to the motors brsve_base.cm strrrag (acainttactioon,,, vait=frue)
32
33
34
(action,
Read sensors to get current state of the robot. (left_n_angle, right_m_angle) (left_motor angle right_motor. angle())
37
38
(pitch, roll) iub.imu.tile dist = sensor distance
41
Send the current state back to the environment. out out_msg "Ifffff", left_m_angle, right_m pitch, roll, dist
42
msg ustruct. pack
45
stdout. buffer. write
Listing 2: Client script example.
A.4 Communication Frequency
Figure 7 offers direct performance comparison of the DroQ agent on the Walker-vo task at communication frequencies of 11Hz and 2Hz. Interestingly, the agent operating at 2Hz shows quicker and more stable convergence. We suspect that the lower communication frequency functions similarly to 'frame skip', widely utilized technique in reinforcement learning. Frame skipping helps to reduce the number of actions an agent takes, thereby simplifying the decision-making processes. This method may explain the more efficient convergence observed with the 2Hz frequency.
A.5 RunAway-v0 Strategies
Figure illustrates the distinct strategies developed by the algorithms. Specifically, Figure shows the final distance measured, while Figure & displays the mean action taken over the entire episode.
20

---

Walker-vO
Alganthmm:
Random
Droc
3Hz
-80
-00
Restard
140
-80
180
20
30
40
60 70
Episodi
Figure 7: Comparison of communication frequencies for the DroQ agent on the Walker-vO task, illustrating the differences between the operational frequencies of 11Hz and 2Hz.
Algorthmms
SAC
DeQ
1800
1400
1200
Distancce
1000
800
-0.4
800
400
-0.8
200
10
20
10
Episode
Figure 8: Final distance and (mean) action taken over one episode for the RunAway-ve task.
A.6
Online Training Parameter
Table 12
displays the hyperparameter used in all experiments.
A.7 Dataset Generation Process
The expert dataset for each robot configuration was generated by training Soft Actor-Critic (SAC) agent solve the respective task and record transitions over 100 episodes on the real robot. The random dataset was created by executing random policy for 100 episodes. For example, the collection process for the Walker robot took about an hour, yielding approximately 10,000 transitions. Details such mean reward, number of transitions, and collection episodes for each dataset can be found in Table TT. The dataset available on Hugging face
A.8 Offline Training Parameter
For the online algorithms, we used the same parameters in our offline rl experiments as in the online experiments.
We trained the models for various tasks and datasets with different update counts 13 task was trained for 2,000 updates on both the expert and random datasets. For the Spinning-vo task, we used 5,000 updates across both datasets. The Walker-vo task required 10,000 updates for both expert and random datasets. Finally, the RoboArm-vO task was trained for 10,000 updates on the random dataset and 5,000 updates the expert dataset.
The RunAway-vo
21

---

Parameter
DroQ
SAC
TD3
Learning Rate (Ir)
10
3 x 10
3 X 10-4 256
Batch Size
256
256
UTD Ratio
20
Prefill Episodes
10
10
10
Number of Cells
256
256
256
Gamma
0.99
0.99
0.99
Soft Update
0.995
0.995
0.995
Alpha Initial
1
Fixed Alpha
False LayerNorm
False
Normalization
None
None
Dropout
0.01
0.0
0.0
Buffer Size
1000000
1000000
1000000
Exploration Noise
0.1
Table 10: Hyperparameter for the agents DroQ. SAC, and TD3
Task
Mean Reward
Expert Transitions
Random Transitions
Episodes
Walker-vO
69.12
9, 244
10, 000
100
RoboArm-v
9.87
1, 297
10, 000
100
RunAway-vu
Spinning-wi V
8981.19
5, 00000
5. 0.000
100
Table 11: Dataset Statistics
A.9 Network Architecutre
Throughout the experiments, all algorithms utilize the same architecture for the policy, Q-functions, and value functions (where applicable). Each network is structured as three-layer multilayer perceptron (MLP), with specific TorchRL actor modules used for the policy, depending on the algorithm. For more details, we refer readers to the code repository: GitHub
The only variation in architecture occurs when incorporating pixel-based observations. In this case, a convolutional neural network (CNN) is used to encode the image data. These encodings are then concatenated with the bensor-based encodings. and the combined embeddings are passed through shared MLP. Detailed implementation specifics can be found in the GitHub repository.
22

---

Parameter
BC
IQL
CQL
Learning Rate (Ir)
x 10
3 x 10
3 x 10 4
Batch Size
256
256
256
Number of Cells
256
256
256
Gamma
0.99
0.99
Soft Update
0.995
0.995
Loss Function
L2
L2
L2
Temperature
1.0
1.0
Expectile
0.5
Min Q Weight
1.0
Max Q Baakuu
False
Deterministic Backup
False
Num Random Actions
10
With Lagrange
True
Lagrange Threshold
5.0
Normalization
LayerNorm 0.01
None
None
Dropout
0.0
0.0
BC Steps
1,000
Table 12: Hyperparameter for the agents BC, IQL, and CQL
Task
Expert
Random
RunAway-
2,000
2,000
Spinning-vO
5,000
5,000
Walker-vO
10,000
10,000
RoboArm-v0
5,000
10,000
Table 13: Number of offline training updates for each task and dataset
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are demonstrated our results section and can be further observed in the evaluation videos provided on project page.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations of the work, including the low communica- tion speed and its causes, the lack of millimeter -precieer-prreeeeee constructions, backlash, and noisy sensors. It also highlights how the limitation of missing vision sensors can be extended.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
24

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not provide theoretical results; instead, focuses on demon- strating practical applications.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: For reproduction, robotic hardware is needed; however, the paper fully dis- closes all the information needed to reproduce the main experimental results, including building instructions for the robots, specifications for the environment setup, reward func- tions, and hyperparameter settings for the algorithms. Additionally, full access to the code is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
25

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provides full access to the code and data and provides detailed instructinos to reproduce the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
poolic/guides// CodeSubmissionoolloo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wttp:/ic/ggii////eeeeeeeeeeeeeet httpc//gguideeeeeeeuuitiiiiii www.oodeuuiiiisonnn....
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, the paper specifies all the hyperparameters for the algorithms and the specific settings for each environment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run our experiments over 5 seeds for each algorithm and provide plots displaying the mean, minimum, and maximum values across all experiments. Additionally, the evaluation results are provided over seeds, with results displayed using the mean and standard deviation.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies that regular affordable laptop is used for the experiments, highlighting the low cost running the experiments. Additionally, specific training times are provided in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurips. http:p/p......pooeoccccecccaeeie
Answer: [Yes]
Justification: The research conducted in the paper conforms in every respect with the NeurlPS Code of Ethics. We do not foresee any potential harms or negative social impact arising from our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
27

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We do not discuss negative societal impacts, we do not foresee any harmful or negative consequences from training LEGO robots. There is no risk of disinformation. generating fake profiles, surveillance, or similar issues. However, the paper focuses on the positive societal impact by democratizing robotics and reinforcement learning for research and education.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no risk of possible misuse of models or data described in the paper, so specific safeguards are necessary for their release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
28

---

Answer: [Yes]
Justification: Yes, prior work on which this paper builds is properly mentioned and cited, and the creators or original owners of assets used in the paper are appropriately credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented. We provide building plans for the robots used in the experiments and access to the code for training them. Additionally, we offer templates for creating training environments and client scripts to train new custom robots.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
29

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "xJGnwKjuXp", "reviewer": "Reviewer_6ArU", "summary": "This paper demonstrates that cheap LEGO robots can be trained in under 120 mins on a laptop to perform simple tasks using sim2real approaches. The paper is motivated by making robot learning research more accessible for educational settings. A software framework called BricksRL is released in open source that integrates PyBricks and TorchRL with Gym environments for controlling modular LEGO-based components, motors and sensors.", "questions": "- The paper is currently lacking details on policy architectures used for training. Please add.\n- Are any of the policies vision-based?  As the number of parameters increases, the training time will also increase. This will stress the reliability of LEGO hardware - possibly breakages that may complicate learning on more complex tasks. Some expanded discussion on this would be helpful.", "limitations": "Yes, to some extent. One issue with low-cost robots is easy wear and tear precluding large-scale data collection. The presence of backlash, noisy sensing may also limit experimentation for more precise tasks. Please expand more on limitations in this regard.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is clearly written, with a strong motivation section and a sound implementation of RL algorithms.\n* The paper is accompanied with instructions on how to build LEGO robots paired with BrickRL for exploring RL/control/robot learning algorithms much more cheaply than with industrial robots.\n* The finding that simple tasks can be trained for in a couple of hours on a normal laptop is compelling for educational experimentation.\n* LEGO robots are easily extensible, so one may envision quickly designing higher-degree-of-freedom robots for research", "weaknesses": "* The core contributions of this paper are not technically novel - there are no new algorithms introduced, and BricksRL is a standard Gym framework for RL training. \n* The tasks demonstrated are relatively simple (walking, spinning, reaching).  Hence, it is not clear how much value this framework provides beyond educational experimentation in simulation. \n* The RL algorithms benchmarked may not reflect current state of the art approaches - but this is not the prime focus of this paper. The variance in the plots seems to be extremely high.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response Weaknesses:\n\n- Our primary contribution lies in providing accessible, low-cost hardware integration for real-world robotics experimentation. Our goal is to lower the barrier to entry for robotics research and education, enabling a wider audience to engage with physical robots beyond simulations. LEGO parts are widely available, robust, and reproducible because they are mass-produced. We are not aware of any other robotic platform with such characteristics, making this paper a first instance of that.\n\n- The demonstrated tasks serve as proof-of-concept examples to validate the framework's functionality. It is not at all obvious that LEGO can be used for RL tasks, as it is not designed for that. Gears have generous tolerance, the stepping motors offer encoders but the precision is about 1 degree, and structurally, they are made of bricks. In this paper, we show that all this hardware can be used effectively for RL. \n\n- Our focus is on demonstrating the framework's compatibility with a range of algorithms rather than pushing the boundaries of RL performance given that this is a totally new platform. The outcome is a validated platform for the democratization of robotics. However, we made sure to implement the RL framework using the official RL library from PyTorch. This allows direct usage of all state-of-the-art RL algorithms. Since submission, we have already expanded our example experiments to include several offline RL algorithms, such as BC, TD3+BC, IQL, and CQL, trained on collected datasets. We've also integrated pre-trained foundation models like VIP transform from TorchRL. Which can be used to give reward signals to sparse reward tasks but can also be utilized to label hand-collected datasets that have no reward signal. VIP can enable simple, few-shot offline RL on real-world robotic tasks for low data quantities. We are making available these examples and the offline datasets with the revised version of the paper.\n\n- The high variance in the plots is potentially stemming from the noisy nature of real-world robotics. This variability itself is an interesting area for future research, investigating robust learning methods in noisy, real-world environments.\n\nResponse Questions:\n\nWe apologize for the lack of details on policy architectures in the current version of the paper. We will add this information in the appendix. All examples are available in full detail in the GitHub repository. Our default architecture for both policies and Q-functions is a multilayer perceptron (MLP) with three linear layers and ReLU activation functions. We also incorporate dropout and layer normalization specifically for the DroQ algorithm. \n\n- For the vision-based environment (roboarm-mixed-v0), we employ a hybrid architecture. This combines image observations with state observations of the robot angles. Specifically, image observations are processed through a convolutional neural network (CNN), while state observations are encoded using an MLP. The resulting image and state embeddings are concatenated and passed through a final output head. We have added a detailed description of this architecture in the appendix of the paper.\n\n- Regarding the concern about increased training time and potential stress on LEGO hardware. We have run the same LEGO components for almost two years to prepare this paper with zero faulty parts. LEGO parts are created for kids and mass-produced so they don't break. \n\nResponse Limitations:\n\nThank you for raising these important points about the limitations of low-cost robots. The main limitations of LEGO components are in the tolerance of the gears, precisions of the encoders, and torque of the motors. However, in our experiments, we have shown that BricksRL is capable of working around those. Our extensive testing and experiments with the LEGO robots have yielded very robust results. Throughout our trials, we did not encounter a single failure of any motor or gear, which speaks to the reliability of these components even under repeated use.\nRegarding backlash and sensor noise, these factors are certainly present in our setup. However, their impact on policy learning and execution has not been significant. This is evidenced by our successful sim2real experiments, where policies trained in idealized simulations were directly transferable to real robots with minimal issues. Our approach of adding small amounts of noise to simulated observations to mimic backlash and sensor inaccuracies proved effective in bridging the reality gap.\nThe architecture ensures that the computational load of training and inference remains separate from the physical robot, mitigating concerns about hardware reliability or potential breakages due to increased model complexity. The LEGO components are primarily involved in the physical execution of actions and collection of observations, tasks which are not affected by increases in model size. We have added this information to the main manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U1S4oflCVO", "reviewer": "Reviewer_R3sL", "summary": "This work proposes a flexible and cost-effective platform named BricksRL with LEGO builds aiming to lower the cost of RL research and education. Experiments also demonstrated that LEGO robots can be trained within 120 minutes on normal computers to achieve simple tasks such as moving, walking, and grasping. An offline dataset is also provided for offline RL or imitation learning.", "questions": "- In what way can LEGO robot models be converted to models that can be used in simulation? \n- Does BricksRL also support urdf model construction or the integration into mainstream simulators like Mujuco or Isaac Sim/Gym?", "limitations": "As mentioned in Weaknesses, the scientific contribution of this work is limited.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Practical utility has been demonstrated by showing the deployment results of RL methods on three LEGO robots such as SAC, TD3, and DroQ.\n- This LEGO robot platform is low-cost and flexible with clear instructions and open-source code.", "weaknesses": "- BricksRL’s value as an educational platform is clear but the scientific contribution is limited because there's no innovation proposed on algorithms.\n- A dataset constructed with BricksRL may be of greater value for offline reinforcement learning and imitation learning.\n- The tasks that BricksRL supports are relatively simple, such as walking, spinning, and reaching.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response Weaknesses:\n\nThank you for your thoughtful feedback, we appreciate your recognition of its practical utility and educational value. However, we respectfully disagree with the assessment that the scientific contribution is limited. \n\nWe would like to emphasize several key points that demonstrate the significant scientific value of BricksRL:\n- BricksRL is the first platform to seamlessly integrate state-of-the-art reinforcement learning algorithms with hardware that is economical, modular, mass-produced, and easy to use like LEGO. This allows everybody in the world to focus on RL algorithms but test their ideas on real robots straightaway. There are over 1M LEGO hubs sold worldwide which are compatible with Pybricks and BricksRL as of today. We cannot emphasize enough how transformative this aspect is in the field of AI for robotics.  \n- By enabling the creation of low-cost, customizable robots, BricksRL addresses a critical challenge in RL research - the ability to conduct reproducible experiments on multiple real robots at once. This scalability and reproducibility is crucial for advancing RL algorithms in real-world settings.\n- We have shown with BricksRL that complex RL algorithms like SAC, TD3, and DroQ can be successfully applied to train LEGO robots end-to-end in the real world within reasonable time frames. This is a necessary validation and demonstration of the platform towards making real-world RL more accessible and practical.\n- We have also demonstrated the ability to incorporate non-LEGO sensors to increase its potential for broader applications in robotics research, beyond just LEGO components. LEGO itself might provide more sensors in the future.\n- By providing built plans and training results, BricksRL contributes to the reproducibility of RL experiments in robotics, a crucial aspect of scientific research often challenging in real-world settings. BricksRL has clear educational benefits, but we have also created it for its research potential. We believe that BricksRL's contribution represents a significant step towards democratizing real-world RL research in robotics, addressing key challenges in the field such as cost, scalability, and the reality gap.\n\nOffline datasets: \n\nWe have already generated valuable offline datasets with our LEGO robots. As part of our revised paper submission, we are making these datasets publicly available in the code repository to facilitate research in offline RL that is directly available in TorchRL. \n\nWe have curated datasets for the three robot configurations: 2Wheeler, Walker, and RoboArm. These datasets encompass both expert and random data, specifically tailored to the tasks explored in our experiments.\n\nFor the 2Wheeler, we offer datasets for the 'RunAway-v0' and 'Spinning-v0' environment tasks. The Walker robot dataset focuses on the 'Walker-v0' task, while the RoboArm robot dataset addresses the 'RoboArm-v0' task.\nThe expert data for each dataset was carefully collected using the following process: We first trained a Soft Actor-Critic (SAC) agent to successfully complete the specific task. Upon achieving satisfactory performance, we evaluated the agent over 100 episodes per task, recording all transitions. For example, the evaluation process took approximately one hour for the Walker robot and yielded 10,000 expert transitions.\n\nTo complement the expert data, we also generated random datasets. These were created by executing a random policy in the environment, similarly for 100 episodes per task. This process resulted in eight distinct datasets:\n1. RunAway-v0-expert\n2. RunAway-v0-random\n3. Spinning-v0-expert\n4. Spinning-v0-random\n5. Walker-v0-expert\n6. Walker-v0-random\n7. RoboArm-v0-expert\n8. RoboArm-v0-random\n\nWe trained successfully offline-RL algorithms, including BC, TD3+BC, IQL, and CQL using these datasets. This allows researchers to compare the performance of various RL algorithms, both offline and online, when trained on expert demonstrations versus random interactions. Such comparisons can provide valuable insights into the effectiveness and adaptability of different RL methods across various data quality levels and robot configurations.\n\n\nResponse Questions:\n\nCurrently, our approach to simulating LEGO robot models in BricksRL is straightforward but effective. In our experiments, we simulated the transition dynamics by directly applying the actions to the simulated motor angles. This works well because our action space represents the delta angles for robot movement, allowing for a simple yet accurate simulation of the robot's behavior.\nBricksRL does not yet support URDF model construction, which would allow for more sophisticated simulations and easier integration with mainstream simulators like MuJoCo or Isaac Sim. However, a few LEGO CAD (https://www.leocad.org/) are actually available. One could build completely virtual robots using only LEGO parts. The complete list of parts is then listed. The LEGO ecosystem is maybe unsurprisingly very rich. It would be therefore time-consuming but relatively straightforward to build simulated LEGO robotics. We are considering this avenue of developing MuJoCo-based LEGO simulations for the future.  This is, however, secondary, to prove that RL research for robotics can be done at all with LEGO which is the outcome of this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rVan9mqqPX", "reviewer": "Reviewer_tHkr", "summary": "This paper presents BrickRL which is a system for using reinforcement learning within the context of lego robotics. The paper provides an overview of their setup, how they used TorchRL and PyBricks to interface with the the lego robots. They also provide results that show the feasibility of this system to use off-policy algorithms to train in the real world.", "questions": "Some of these questions are going to be in the unanswered limitations as well.\n- How long do these robots last? Do the motors wear down during training? Are they expensive to replace?\n- Could you use a wired connection for training? I realize it causes some issues but would it increase training Hz?\n- How much did the setups you used cost?", "limitations": "The authors don't address many limitations in their paper which I do wish they more clearly did. Scattered around are a few limitations addressed like communication frequency but I think that limitations could be more clearly thought of and addressed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Originality: From my undersanding, this is the first paper to combine the lego robotics setup that has been popular in education for a while with RL. They cite other efforts to create low-cost robotics but even those are significantly more expensive than a simple lego setup, which further improves the originalitiy in my mind as a first of its kind very low-cost and flexible setup.\n\nQuality: The main claim the authors make in this paper is their method is able to train agents on lego robots that perform well in the real world which they do. It would have been nice to mix both off and on policy algorithms in your results as well as maybe some offline RL just to clearly show the vercitility of your method. That being said, the claim of \"agents can learn for lego RL\" is clearly answered affermatively (at least for off-policy algorithms) from their results.\n\nSignificance: Major. This paper on the surface might seem simple as it is just combining lego with RL but in my opinion the significance of low-cost robotics for RL is massive. Researchers with less funding using this system for their work to test different algorithms on low-cost robots that are extremely easy to repair. The results are good enough that this system is clearly working and has the potential to address many accessibility issues with RL in robotics.", "weaknesses": "Clarity: The paper has no major errors but can be confusing as it does state things that feel like implementation details in the main paper that should likely go in the appendix and did make it harder to read.\nPossible improvements:\n- I think the PyBricksHubClass paragraph feels out of place and a little confusing. This could probably be just a simple \"PyBricks provides a class that...\" and simply state the benefits. I don't think the fact that it uses BLE is important.\n- The BaseEnv paragraph also feels out of place. I think that putting it in there and saying that you can use a BaseEnv to create custom environments is redundant. The real benefit here is the second paragraph about enviornment transforms and other models.\n- The useage of TensorDict feels just like an implementation detail, while it might be critical, these three paragraphs in 3.2 all feel like they could be one or two sentences and then you can move on and save more room for results.\n- Simlar comment for Client Script and Robot-Environment Communication. It seems just like an implementation detail that might even be completely ok in the appendix. But Communication Speed is a very interesting section that I really like (addresses a possible limitation).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response Weaknesses:\n\nWe greatly appreciate your valuable input and suggestions for improving the clarity of our paper. We are pleased to incorporate the proposals as far as possible in the final version of our paper. \n\n\nResponse Questions:\n\nWe're happy to provide more details about our LEGO robot setups:\n\nRobot longevity and motor wear:\n- Battery life varies depending on the robot's complexity and usage intensity. For example, our walker and RoboArm models typically operate for 3-4 hours on a single charge, but it is possible to directly provide power via a USB connection to a larger battery pack.\n- Static robots like the RoboArm can be directly connected to a power supply via a USB for extended operation in training and execution\n- Throughout all our experiments and tests over a couple of years, we haven't experienced any part failure, indicating good durability. If replacement is needed, individual motors cost between 40-60€.\n\nWired connections for training:\n- This is possible for stationary robots using the USB connection. \n- We recognize the potential benefits of increased communication speed. PyBricks is already experimenting with better communication channels and they will be made available in BricksRL immediately.\n\nCost of setups:\n- Our setups utilize components that can be largely found in the LEGO Education kit and extension kit.  The current price range for these sets is approximately 700€.\n\n\n\nResponse Limitations:\n\nThe main limitations of LEGO components are in the tolerance of the gears, precisions of the encoders, and torque of the motors. However, in our experiments, we have shown that BricksRL is capable of working around those. Our extensive testing and experiments with the LEGO robots have yielded very robust results. \n\nWe further show the robustness in the sim2real experiments, where policies trained in idealized simulations were directly transferable to real robots with minimal issues. Our approach of adding small amounts of noise to simulated observations to mimic backlash and sensor inaccuracies proved effective in bridging the reality gap.\n\nRegarding the limitation of communication speed, as mentioned above Pybricks is working on better communication channels, and once available those will be immediately included in BricksRL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4paRzLurdm", "reviewer": "Reviewer_Tpu8", "summary": "The paper presents a framework for training reinforcement learning policies on LEGO robots in the real world.\nBricksRL integrates the LEGO robotics hub PyBricks and a reinforcement learning library TorchRL, providing an easy interface to implement and deploy RL algorithms. A robust infrastrure for robot-environment-algorithm communication is implemented for real-world deployment.\nA camera is integrated in the system beyond LEGO’s standard sensor set, which expands the platform’s capabilities.\nExperiments on various tasks on three LEGO robots (2wheeler, walker, and robotarm) are demonstrated, validating the system’s capabilities for real-world robotics applications. Ablations are studied to compare sim2real transfer with learning from the real world, and w/ or wo/ camera sensor inputs.", "questions": "Which simulator is used?\n\nIs RoboArm-mixed-v0 trained only using real-world data? Perhaps because sim2real transfer with image is hard?\n\nIs the trained policy robust to some slight deviations of robot hardware during explorations, such as shifted parts, sensor drifting, and system latency?", "limitations": "Yes, the authors addressed the limitations well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Constructing robots with LEGO parts is low-cost and widely available. This paper demonstrates deploying reinforcement learning on LEGO robot, which is very inspiring for low-cost robot research.\n\nLEGO parts are modular and reusable, allowing the user to tailor robot designs to specific tasks. This provides a nice platform for cross-embodiment policy research, task-specific robot design, etc.\n\nThe paper provides an effective solution to deploy reinforcement learning algorithms on low-cost robots, by integrating PyBricks and TorchRL, and implementing a robot-emvironment-algorithm communication via bluetooth connection. The system is proved to be robust by deploying real-world reinforcement learning policies.\n\nDetailed experimental results on three LEGO robots are demonstrated, showcasing the system’s capability to train real-world rl policies on low-cost robots.", "weaknesses": "Integrating LEGO hub with BricksRL, due to communication overhead, the system frequency is only 11 Hz, which limits the capability of the system to perform certain tasks such as dynamic manipulation tasks.\n\nIt would be insightful for the authors to provide more discussions and ablations on the robustness of the system. The paper noted that the system has low communication speed, the lack of millimeter-precise constructions, backlash, and noisy sensors. But does the policy learn to be robust to these noises during training? Or are there certain strategies that authors employ to deal with these issues?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response Weaknesses:\n\nThank you for your assessment of the weaknesses, we are happy to address these points to provide more clarity.\n\nWe recognize the potential benefits of increased communication speed and are willing to improve this. We've contacted PyBricks directly to collaborate on enhancing communication rates and they claim to have better protocols almost ready. Once implemented they will be made available in BricksRL immediately. However, we hardly had any problems due to the slower communication speed. As described in the paper, lower communication even led to more stable and faster learning for the walker robot. \n\nThe policies learnt were robust notwithstanding existing backlashes and noise. For the real-world training, we did not take any measures to stabilize the learning as it was not necessary. However, as mentioned in the paper, for the sim2real transfer with the policies trained in simulations we added noise to the actions to simulate backlashes and sensor noises. This was sufficient to learn stable policies that could be evaluated successfully in the real world. \n\nResponse Questions:\n\n- The robots were mainly trained online in the real world. We only developed two simulations to demonstrate the sim2real capabilities. Our approach to simulating LEGO robot models in BricksRL is straightforward but effective. In the two simulation environments, we simulate the transition dynamics by directly applying the actions to the simulated motor angles. This works well because our action space represents the delta angles for robot movement, allowing for a simple yet accurate simulation of the robot's behavior.\n- Yes, the RoboArm-mixed-v0 is fully trained in the real world, we did not develop any simulator for this environment. The fact that LEGO can be used in this setting proves the value of the LEGO platform for RL research.\n- Yes, the trained policies are robust to different sensor noises and backlashes. Different sensor noises/deviations are more harmful than others. For example, we did notice that during training of the RoboArm-mixed-v0 with image inputs lightning conditions are very important and need to be stable during the whole training. However, this is more specifically for the contour detection of the ball for reward calculation and not so much for policy learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have added this subsection (PDF) to the paper.

Meta-review: The reviewers and the AC were impressed by the idea and system setup (software + hardware). Thanks for making the data available. The system has been extensively tested. The rebuttal managed to alleviate all major doubts of the reviewers.I believe this is a very interesting and promising piece of work that provides a low-cost benchmark for real-robot RL experiments with a large potential for customization and extensions.
Decision: Accept (spotlight)