RAN Simulator Is NOT What You Need: O-RAN Reinforcement Learning for the Wireless Factory Ta Dang Khoa Le ta-dang-khoa.le@eurecom.fr EURECOM Biot, France Navid Nikaein navid.nikaein@eurecom.fr EURECOM Biot, France ABSTRACT As modern manufacturing lines embrace greater modularity and flexibility, the need to transition factory networks from wired to wireless grows. Yet the mission-critical nature of factory networks poses a key challenge - connecting numer- ous diverse machines with high QoS predictability. After formulating this challenge as predictable RAN optimization via Reinforcement Learning (RL), we highlight a major-yet- overlooked modeling issue: matching the packet handling mechanics of a production/real RAN software. In this paper, we show that these mismatches inside RAN simulators can cause non-trivial QoS gaps in production. Then, we present Twin5G, a novel training solution that brings scalable and near-discrete-time emulations to real RAN software, remov- ing the need for RAN simulators. In a RAN Slicing example, Twin5G-trained policy outperforms simulator-trained and standard RL-trained policies in both QoS achieved (+16%) and predictability (+19%) during tests. ACM Reference Format: Ta Dang Khoa Le and Navid Nikaein. 2023. RAN Simulator Is NOT What You Need: O-RAN Reinforcement Learning for the Wireless Factory. In The 29th Annual International Conference on Mobile Computing and Networking (ACM MobiCom ’23), October 2–6, 2023, Madrid, Spain. ACM, New York, NY, USA, 3 pages. https://doi.org/ 10.1145/3570361.3615758 1 INTRODUCTION To meet increasingly customized and smaller manufactur- ing orders, modern manufacturing lines are designed to be highly modular and reconfigurable. This necessitates the use Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ACM MobiCom ’23, October 2–6, 2023, Madrid, Spain © 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-9990-6/23/10...$15.00 https://doi.org/10.1145/3570361.3615758 of mobile machinery at minimal rewiring costs, i.e., wire- less networking [1]. However, transitioning mission-critical factory networks from wired to wireless is a challenge; one needs to connect as many UEs as possible, despite much limited resources, while guaranteeing each UE’s QoS before actual deployment (i.e., QoS predictability), despite diverse requirements and stochastic workload variations. In this paper, we formulate this wireless factory challenge as (a) per-UE predictable QoS optimization solved with (b) RL-based RAN control. (a) is crucial because we want to maximize the value of each used resource unit. Yet, optimiz- ing system-level or multi-UE objectives cannot guarantee predictable network conditions for any UE [2]. (b) is neces- sary because, due to radio scarcity, RAN remains the only bottleneck in modern network deployments [3]. Yet, opti- mizing it requires a sequential stochastic control approach to handle adaptive traffic profiles and partially-observable channel variations. Under this formulation, multiple per-UE RL policies simultaneously control the RAN, with each pol- icy’s expected total reward being the prediction for its UE’s in-production QoS. Still, as Deep RL is sensitive to modeling errors [4], high QoS predictability requires accurate emulation of produc- tion traffic, channel, and RAN-stack dynamics. In this paper, we leave the well-known traffic-channel modeling errors to future work, and fo