# Multi-Agent Contextual Combinatorial Multi-Armed Bandits with Linear Structured Super Arm: application to energy management optimization in Smart Grids

Eloann Le Guern-Dall, Raphaël Féraud, Guy Camilleri, Patrick Maillé, H.

Ben Ahmed, Juan J Cuenca, Riadh Zorgati, Fabien Petit, Anne Blavette

► To cite this version:

Eloann Le Guern-Dall, Raphaël Féraud, Guy Camilleri, Patrick Maillé, H. Ben Ahmed, et al.. Multi-Agent Contextual Combinatorial Multi-Armed Bandits with Linear Structured Super Arm: application to energy management optimization in Smart Grids. PGMODAYS 2024, Nov 2024, Palaiseau, France. hal-04921494

HAL Id: hal-04921494

https://hal.science/hal-04921494v1

Submitted on 30 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Multi-Agent Contextual Combinatorial Multi-Armed Bandits with Linear Structured Super Arm: application to energy management optimization in Smart Grids

Eloann Le Guern-Dall'o<fnref n="1" /> Raphaël Féraud<fnref n="2" /> Guy Camilleri<fnref n="3" /> Patrick Maillé<fnref n="4" />

Hamid Ben Ahmed<fnref n="5" /> Juan Cuenca<fnref n="1" />,<fnref n="3" /> Riadh Zorgati<fnref n="6" /> Fabien Petit<fnref n="7" /> Anne Blavette<fnref n="1" />

**Keywords:** Smart grid, Renewable Energy, Multi-Agent, Reinforcement Learning, Bandits

The integration of electric vehicles (EVs) and renewable energy sources (RES) into future electrical grids presents both significant challenges and opportunities for energy management. While RES production is an incentive for increased demand at specific times of the day (e.g., at noon for photovoltaic production), the distribution system operator (DSO) should also prevent grid congestion. The optimal strategy for integrating each EV with regard to the DSO constraints in a manner that maximizes the local use of RES is an NP-hard problem, as it requires the resolution of a mixed-integer linear programming problem (MILP) [1]. Moreover, due to weather and human behaviors, this optimization should be done under uncertainty. In particular, learning algorithms from the family of Bandit algorithms have been studied in [2], [3] to handle uncertainty. In light of the decentralized algorithms developed in [2], we propose a Multi-Agent Contextual Combinatorial Multi-Armed Bandits (MA-CC-MAB) approach with linearly structured super arms. By leveraging contextual information such as time-varying renewable energy generation, grid load conditions and day of the week, each agent will selfishly and dynamically schedule its charging intervals while considering grid constraints. The use of linearly structured super arms enables efficient exploration and exploitation in a high-dimensional combinatorial action space, while decentralization will address scalability issues inherent to large systems. The use of contextual information allows adapting to different conditions of the environment, ensuring efficient decision-making under uncertainty. We show performance of this algorithm on the IEEE LVTF network model (55 agents) with PV production, using pandapower python library for the load flow simulation. This work is conducted as part of the EDEN4SG project funded by the French National Agency for Research (ANR), under grant agreement ANR-22-CE05-0023.

## References

[1] S. Zafar, V. Maurya, A. Blavette, G. Camilleri, H. B. Ahmed, and M. Gleizes, "Adaptive Multi-Agent System and Mixed Integer Linear Programming Optimization Comparison for Grid Stability and Commitment Mismatch in Smart Grids," in 2021 IEEE PES Innovative Smart Grid Technologies Europe (ISGT Europe), Oct. 2021, pp. 01-05.

[2] S. Zafar, R. Féraud, A. Blavette, G. Camilleri, and H. B. Ahmed, "Multi-Armed Bandits Learning for Optimal Decentralized Control of Electric Vehicle Charging," in 2023 IEEE Belgrade PowerTech

[3] M. Brégère, P. Gaillard, Y. Goude, and G. Stoltz, "Target Tracking for Contextual Bandits: Application to Demand Side Management," in Proceedings of the 36th International Conference on Machine Learning, PMLR, May 2019, pp. 754-763.

<footnote n="1">¹UnivRennes, CNRS, ENS Rennes, CentraleSupelec, Nantes Université, IETR UMR 6164, Rennes, France, eloann.le-guern-dallo, anne.blavette@ens-rennes.fr, juan.cuenca@centralesupelec.fr</footnote>

<footnote n="2">²Orange Labs, Lannion, France, raphael.feraud@orange.fr</footnote>

<footnote n="3">³IRIT-SMAC, Univ. de Toulouse, CNRS, Toulouse INP, UT3, Toulouse, France, guy.camilleri@irit.fr</footnote>

<footnote n="4">⁴IMT-Atlantique, Univ. de Rennes, Rennes, France, patrick.maille@imt-atlantique.fr</footnote>

<footnote n="5">⁵UnivRennes, ENS Rennes, CNRS, IETR UMR 6164, Rennes, France, hamid.benahmed@ens-rennes.fr</footnote>

<footnote n="6">⁶EDF Lab Paris-Saclay, Palaiseau, France, riadh.zorgati@edf.fr</footnote>

<footnote n="7">⁷SRD, Poitiers, France, fabien.petit@srd-energies.fr</footnote>