6G BRAINS Deliverable D2.3 Multi-agent Deep Reinforcement Learning Scheme Specifications

This deliverable reports all the activities and outcomes related to specifying the requirements for the multi-agent Deep Reinforcement Learning (MA-DRL) scheme of 6G BRAINS. Several applications of DRL are identified by the partners in relation to the general 6G BRAINS use cases defined in D2.1. For each of the DRL applications, it will identify the interfaces between the network and the MA-DRL, as well as the data that need to transit between the MA-DRL scheme and the network. This report lays the ground for the developments of the RL-applications in the MA-DRL scheme. Ref. Ares(2021)5954880 30/09/2021 6G BRAINS H2020-ICT 101017226 Deliverable D2.3 Page 2 of (83) © 6G BRAINS consortium 2021 Disclaimer This document contains material, which is the copyright of certain 6G BRAINS consortium parties, and may not be reproduced or copied without permission. In case of Public (PU): All 6G BRAINS consortium parties have agreed to full publication of this document. In case of Restricted to Programme (PP): All 6G BRAINS consortium parties have agreed to make this document available on request to other framework programme participants. In case of Restricted to Group (RE): All 6G BRAINS consortium parties have agreed to full publication of this document. However this document is written for being used by <organisation / other project / company etc.> as <a contribution to standardisation / material for consideration in product development etc.>. In case of Consortium confidential (CO): The information contained in this document is the proprietary confidential information of the 6G BRAINS consortium and may not be disclosed except in accordance with the consortium agreement. The commercial use of any information contained in this document may require a license from the proprietor of that information. Neither the 6G BRAINS consortium as a whole, nor a certain part of the 6G BRAINS consortium warrant that the information contained in this document is capable of use, nor that use of the information is free from risk, accepting no liability for loss or damage suffered by any person using this information. The EC flag in this document is owned by the European Commission and the 5G PPP logo is owned by the 5G PPP initiative. The use of the EC flag and the 5G PPP logo reflects that 6G BRAINS receives funding from the European Commission, integrated in its 5G PPP initiative. Apart from this, the European Commission and the 5G PPP initiative have no responsibility for the content of this document. The research leading to these results has received funding from the European Union Horizon 2020 Programme under grant agreement number 101017226 – 6G BRAINS – H2020-ICT2020-2. Deliverable D2.3 6G BRAINS H2020-ICT 101017226 © 6G BRAINS consortium 2021 Page 3 of (83) Impressum [Full project title] Bring Reinforcement-learning Into Radio Light Network for Massive Connections [Short project title] 6G BRAINS [Number and title of workpackage] WP2 User/Learning Requirements and System Architecture [Number and title of task] T2.3 Multi-agent Deep Reinforcement Learning Scheme Specifications [Document title] Multi-agent Deep Reinforcement Learning Scheme Specifications [Editor: Name, company] Alexandre Kazmierowski & Victor Gabillon, TSG [Work-package leader: Name, company] Yue Zhang, ULEIC [Estimation of PM spent on the Deliverable] 16 PMs Copyright notice © 2021 Participants in 6G BRAINS project 6G BRAINS H2020-ICT 101017226 Deliverable D2.3 Page 4 of (83) © 6G BRAINS consortium 2021 Executive summary One of the main objectives of 6G BRAINS is to bring Reinforcement Learning (RL) into Radio Light Network for massive connections. Radio Light Network covers a wild variety of components and use cases which leads to many industrial scenarios where machine learning approaches such as RL are expected to bring improved performance. Similarly, RL is itself composed of many learning schemes and algorithms. In order for all the partners to coordinate their efforts the 6G BRAINS project will build a Multi-Agent Deep Reinforcement Learning scheme (MA-DRL) building a common structure and code base. This scheme is a standardised general interface between the diverse industrial scenarios brought by the partners and the various RL algorithms and learning schemes. This document presents the MA-DRL scheme in order to give a common basis for the deployment of the RL in the 6G BRAINS applications. This is implemented in three stages. • Each partner presents his or her industrial scenario and how it can be formulated as an algorithmic problem. While the state-of-the-art techniques will be discussed, Reinforcement Learning is examined as a way to tackle the current algorithmic challenges. • The generic architecture of the MA-DRL scheme is presented decomposing the RL software solution into core components. This architecture has been designed in order to enable multiple schemes of learning and to accommodate any problem formulated as a Markov Decision Process (MDP). • The partners specify how they will implement the different components of the MADRL within their own RL industrial scenarios. Deliverable D2.3 6G BRAINS H2020-ICT 101017226 © 6G BRAINS consortium 2021 Page 5 of (83) List of authors Company Author Contributions ULEIC Yue Zhang, Lu Ge Chapter 3, 5, 6 UBRUNEL John Cosmas, Geoffrey Eappen, Ben Meunier, Kareem Ali Chapter 3, 5, 6 REL Israel Koffman Chapter 3, 5, 6 TSG Alexandre Kazmierowski, Victor Gabillon Chapter 1,2,4,7 BOSCH Alexander Artemenko, Uwe Wostradowski Reviews and Abbreviations and general contributions ECOM Ta Dang Khoa Le Chapter 3, 5, 6 6G BRAINS H2020-ICT 101017226 Deliverable D2.3 Page 6 of (83) © 6G BRAINS consortium 2021 Table of Content List of authors . 5 Table of Content . 6 List of figures and tables . 8 Abbreviations . 9