# Multi-Agent Multi-Armed Bandits: application to EVs smart charging with grid constraints

Eloann Le Guern-Dall'o, Raphaël Féraud, Guy Camilleri, Patrick Maillé, Fabien Petit, Riadh Zorgati, Hamid Ben Ahmed, Anne Blavette

► To cite this version:

Eloann Le Guern-Dall'o, Raphaël Féraud, Guy Camilleri, Patrick Maillé, Fabien Petit, et al. Multi-Agent Multi-Armed Bandits: application to EVs smart charging with grid constraints. PowerTech 2025 - 16th conference, themed Powering the Energy System Transformation, IEEE, Jun 2025, Kiel, Germany. pp.1-6, 10.1109/PowerTech59965.2025.11180603. hal-05262742

HAL Id: hal-05262742

https://hal.science/hal-05262742v1

Submitted on 16 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Multi-Agent Multi-Armed Bandits: application to EVs smart charging with grid constraints

Eloann Le Guern-Dall'o

Univ. Rennes, ENS Rennes, CNRS, IETR lab
Rennes, France

eloann.le-guern-dallo@ens-rennes.fr

Raphaël Féraud

Orange Innovation
Lannion, France

raphael.feraud@orange.fr

Guy Camilleri

SMAC team

IRIT, Univ. Toulouse, CNRS, Toulouse INP
Toulouse, France

guy.camilleri@irit.fr

Patrick Maillé

IMT Atlantique, Univ. Rennes
Rennes, France

patrick.maille@imt.fr

Fabien Petit

SRD

Poitiers, France

fabien.petit@srd-energies.fr

Riadh Zorgati

EDF

Palaiseau, France

riadh.zorgati@edf.fr

Hamid Ben Ahmed

Univ. Rennes, ENS Rennes, CNRS, IETR lab
Rennes, France

benahmed@ens-rennes.fr

Anne Blavette

Univ. Rennes, ENS Rennes, CNRS, IETR lab
Rennes, France

anne.blavette@ens-rennes.fr

**Abstract**—Electrification of energy uses with renewable sources is a major lever of decarbonization. To maximize the use of renewable energy, and tackle its variability and uncertainty, flexible entities such as electric vehicles can be used. This paper presents a scalable, decentralized multi-agent system, where each electric vehicle (EV) seeks the best instants to charge to satisfy its mobility needs, while favoring photovoltaic energy and avoiding congesting the electric network. Each EV makes autonomous decisions using information from its environment, using contextual multi-armed bandit algorithms based on Thompson sampling. Four types of bandit-based algorithms are compared to a baseline through simulations with 55 homogeneous EV agents, achieving a distance of only 14% from an optimal omniscient centralized algorithm. The system's real-world practicality is demonstrated through minimal computational requirements (200 µs per EV per timestep) and limited information sharing, maintaining user privacy while effectively managing grid congestion.

**Index Terms**—smart grid, multi-agent, reinforcement learning, bandits, electric vehicle smart charging

## I. INTRODUCTION

In order to combat climate change, a large number of countries have set ambitious targets regarding the integration of renewables in their energy mixes. This translates into an expected large penetration of photovoltaics (PV) in power systems. However, PV production presents a peak of production around noon which may coincide with a lower power consumption, thus requiring measures to integrate this excess of power. Storing this energy in electric vehicles (EVs) appears as a promising solution. During PV production peak, EVs may be idle and connected to the electrical network on office

premises, and therefore available for charging. However, contributing to maintaining the production/consumption balance at the national/regional level should not come at the cost of creating congestion at the local level. Hence, it is necessary to control these fleets of EVs, which are expected to be of considerable size. Based on a moderate penetration of renewables (scenario "N2" provided by the French transmission system operator [1]), the amount of excess power during summer time at noon in France may indeed reach the equivalent of several millions of EVs charging at a domestic rate of 6 kVA. Considering that it may be expected that only few actors in the market, e.g., aggregators, may be able to provide controlling services, this would result in these actors having to control extremely large fleets of EVs. In consequence, it is necessary to develop scalable methods able to respond to real-time uncertainties (e.g., grid congestion). In this perspective, several references have considered decentralized methods, among which reinforcement learning (RL) has gained a lot of attention in the last decade. RL allows learning agents (such as EVs) to learn the best actions based on their observations from a dynamic and uncertain environment. Among RL approaches, bandits represent a simple but efficient tool when a played action does not change the reward distributions of actions. Under these specific conditions, bandit methods can converge faster in cold start than other RL methods based on Markov decision processes (MDP), while the computing resources they require is usually lower. Last but not least, bandits may provide theoretical guarantees for some problem settings that could enhance the trust of energy actors. Although widely studied and applied in the industry (e.g. recommendation systems [2], telecommunications [3], Cloud auto-scaling [4]...), it is only recently that some works have started emerging on the appli-

<footnote>This work has been carried out as part of the EDEN4SG project (grant agreement ANR-22-CE05-0023) funded by the French National Agency for Research (ANR) which is gratefully acknowledged.</footnote>
cation of bandits to power system problems such as [5]–[9]. Some of the works consider the energy provider perspective, for instance for electricity price design, independently of electrical grid constraints [7]–[9]. Other works are more similar to ours, as they consider the perspective of consumers, or groups of consumers, facing a price fixed by another actor (energy provider and/or aggregator), and willing to minimize their energy bill [5], [9]. While [5] considers the bandit algorithm called EXP3 and proposes variants, [9] proposes a variant of the UCB algorithm [11]. Note however that none of those references consider network congestion. A related approach is taken by Zhao et al. [6], who consider limited power equipment capacities, in an economic dispatch problem with demand-response managed with the UCB algorithm. Unlike most of these works, the method presented in this paper prevents congestion through a mechanism providing guarantees and is inspired by previous work in [12], with key differences in the implementation of bandits algorithms and network formulation, with three novel algorithms using Thompson Sampling and heterogeneous contextual information through linear Thompson Sampling [13]. Although these methods enable a multi-agent scalable and decentralized online learning without environment knowledge and guarantee on prevention of congestion with the aforementioned congestion mechanism, it is important to note that this modifies the original Thompson sampling setting, as the actions may have a significant influence on the distributions of rewards.

The remainder of this paper is organized as follows: Section II details the methodology and models used, Section III presents the case study, while Section IV describes the results. Finally, Section V concludes and presents future research directions.

## II. METHODOLOGY AND MODELING

### A. General problem formulation

The problem considered in this paper focuses on smart charging in the presence of renewables in a work environment where EV owners arrive in the morning and depart in the evening. In the meantime, their EVs are connected to the grid and may charge, with the objective of minimizing their electricity bill for charging (and therefore choose the cheapest instants to charge). They also try to reach a desired state-of-charge before departure and avoid contributing to grid congestion.

The scenario includes an electricity provider, a distribution system operator (DSO), an aggregator, and electric vehicles. The aggregator sends to EV owners some real-time congestion signals that it receives from the DSO. It may also send day-ahead electricity prices from the energy provider. Each EV is a learning agent that will learn online the best instants to charge based on the signal sent by the aggregator (electricity price and congestion signals). Fig. 1 describes the architecture of the system.

The electrical network consists of I electric vehicles connected to a transformer. A simple network (lossless and without reactive power) is considered, where the active power

<figure><img src="image_2.png" /><figcaption>Fig. 1. System architecture. Bold lines are physical connections while arrows are network communications. The aggregator receives all relevant signals from the system and then send signals to influence EVs behavior. The bidirectional arrows indicate that EVs receive price and congestion signal from aggregator, while aggregator receives instantaneous charging power from EVs.</figcaption></figure>

flow at the transformer is equal to the sum of the active powers drawn from each electric vehicle. Each electric vehicle $i \in \{1, ..., I\}$ has a rated power $P_i^{\max}$ and a capacity $E_i^{\max}$. No vehicle-to-grid (V2G) is considered here. At each discrete instant $t$, is denoted by $E_i(t)$ the amount of energy in the battery of each EV $i$, by $P_i(t) \in [0, P_i^{\max}]$ its instantaneous charging power, and its state-of-charge $SoC_i = \frac{E_i(t)}{E_i^{\max}}$. With a temporal resolution $\Delta_t$, the energy level $E_i(t)$ evolves as:

$$E_i(t + \Delta_t) = E_i(t) + P_i(t)\Delta_t \quad (1)$$

where the state-of-charge should not exceed $SoC_i^{\max}$.

It is important to mention that to avoid congestion in the electrical network, a specific congestion signal is assumed to be sent in real-time by the aggregator to a subset of the EVs currently charging, and are labeled as responsible for the congestion. This subset is chosen randomly based on a uniform distribution, so that cancelling their charge reduces overall demand below the congestion threshold: the number of contacted EVs is thus the ratio of the excess demand divided by the rated power of 7 kW common to all EVs. The subset may change every time a congestion is detected; EVs receiving the congestion signal instantly stop charging, while the others can still continue charging normally.

In case an EV $i$ has been prevented from charging due to congestion, it must modify its initial charging plan, and may therefore reselect other instants during the day to obtain its desired state-of-charge at departure. At each instant, it computes its $m_i(t)$ remaining required charging instants to reach its desired state-of-charge at departure $SoC_i^*$ as:

$$m_i(t) = \left[ \frac{[SoC_i^* - SoC_i(t)]^+ E_i^{\max}}{P_i^{\max}} \right] \quad (2)$$

with $[\cdot]$ the ceiling function, and $[\cdot]^+ = \max(0, \cdot)$ the positive part.

The electricity price is considered as varying at each instant of the day (each of them separated by duration $\Delta_t$) and highly influenced by PV production, therefore resulting in lower prices during the day, usually around mid-day time. A
normalized electricity price $c_n(t)$ for each instant $t$ of each day $n$ is assumed to be expressed as:

$$c_n(t) = 1 - \frac{P_{PV,n}(t)}{\bar{P}}$$

with $\bar{P}$ an upper bound of the PV production (e.g., the total EV installed power), ensuring that $c_n(t) \in [0, 1]$ for any day $n$ and for any instant $t$ within the day.

## B. Multi-armed bandit formulation

Multi-armed bandits (MAB) fall into the more general theoretical framework of reinforcement learning [11]. The MAB problem consists in finding the best set of arms (actions) between $K$ available arms (i.e., here, charging instants). Each individual arm $k \in [1, ..., K] = [K]$ is associated to a reward $r_k$ which follows an unknown distribution $D_k$. A binary scalar $k_{t_i} \in \{0, 1\}$ is used to denote a base-arm being played at instant $t \in [T]$ by EV $i$ (1 indicating that the EV charges, while 0 indicates that it does not). For greater clarity, subscript $i$ is dropped in the rest of the document and refer for instance to $k_t$ as the base-arm being played at instant $t \in [T]$ by a given EV. Each EV will include one MAB (for combinatorial MABs) or more (for independent MABs). The sequence of base-arms during an iteration (here, a day) $n \in [N]$ is a vector $s_n = [k_{1,n}, k_{2,n}, ..., k_{T,n}]$ such that $s_n \in \{0, 1\}^T$, and represents the sequence of planned charging instants and called a super-arm.

Multi-armed bandits address the exploration/exploitation dilemma: at each iteration, one can indeed choose whether to explore new (potentially better) solutions or to exploit already-identified good solutions. The goal is to manage that trade-off so as to minimize the pseudo-regret $R(N)$ after $N$ iterations, i.e., the difference between the optimal sequence of actions and the actions actually selected, which can be expressed as:

$$R(N) = \sum_{n=1}^{N} (\mu_{s^*} - \mu_{s_n}) \quad (3)$$

with $s^* = \arg\max_{s \in \{0, 1\}^T, ||s||_1 = m(t)} \mu_s$ is the optimal super-arm for any day, $\mu_{s^*}$ its associated average reward, $\mu_{s_n}$ the expected reward of the actually selected super-arm for day $n$, and $m(t)$ is the required charging instants (Eq. (2)). The optimal super-arm refers to the sequence of actions which is optimal (in expectation) and will be applied every day by a given EV.

The key idea of the formulation is inspired by [12]. It consists in discretizing the time during a day into $T$ instants, out of which the multi-armed bandits must learn to select at each instant $t$ the best $m(t)$ instants to charge. Each EV would therefore include one multi-armed bandit (in the combinatorial setting) or several (in the independent setting), as will be described later. The MABs aim at learning the best charging policy (i.e., selecting the best instants to charge) to minimize the EV owner costs, while meeting the desired state-of-charge at departure. Note that since congestions can lead to no charging, EVs would also lead to avoid provoking

congestion on the electrical network. The reinforcement learning theoretical framework implies that the learning agent(s) receive(s) a reward from the environment which depends partly on their actions. The instantaneous reward $r_t$ depends on the electricity price $c_n(t)$ at each instant $t$ and day $n$ and on the reception of a congestion signal. This reward is received by each EV that charges at a given instant $t$ and is expressed as:

$$\begin{cases} r_t = 1 - c_n(t) & \text{if } \text{congestion}(t) = \text{false} \\ r_t = 0 & \text{else} \end{cases} \quad (4)$$

This paper presents the performance of different types of bandits which are listed below, and which will be described in the following sections:

* independent MABs (Algorithm 1)

* contextual independent MABs (Algorithm 2)

* combinatorial MABs (Algorithm 3)

* contextual combinatorial MABs (Algorithm 3)

These algorithms were compared to a simple baseline and an omniscient MILP which are described in the following section.

1) **Baseline:** The baseline algorithm consists in charging as soon as the EV arrives at arrival time $t_i^a$ with a state-of-charge $SoC_i^a$. In case the EV receives a congestion signal (congestion = true), the EV stops charging and tries to charge again at the next instant until reaching the departure time $t_i^b$ or the desired state-of-charge $SoC_i^*$.

2) **Mixed Integer Linear Programming (MILP):** In this benchmark algorithm, the omniscient aggregator controls all EVs to minimize total charging costs with complete knowledge of future variables and is used as a theoretical upper bound.

3) **Independent MABs:** In independent MABs, distribution of rewards of each arm $N(\mu_k, \nu_k)$ is assumed to be independent from the others. This leads to use a Thompson Sampling for each instant $t \in [T]$ of the day. The mean reward $\mu_s$ of super-arm $s$ is the sum of all mean rewards of selected base-arms: $\mu_s = \sum_{t=1}^{T} s(t)\mu_t$, thus $\mu_s = s^T\mu$ with $\mu = [\mu_1, ..., \mu_T]$.

The choice of Gaussian Thompson Sampling is convenient for continuous reward (Eq. 4) [13]. Thus, each base-arm's reward $r_t$ is given by a Gaussian distribution $N(\mu_t, \sigma_t^2)$ and the prior for the unknown parameter $\theta_t$ is also Gaussian with estimated mean value $\hat{\theta}_t$ and variance $\sigma_t^2$ (see Algorithm 1).

It is important to notice that at each time step, an intra-day reselection (line 11) of the super-arm may be performed among an allowed set of super-arms $S(t)$ (line 10), which excludes past instants $s_o$ (line 11) and includes only super-arms with $m(t)$ charging instants (line 9).

4) **Contextual independent MABs:** With the previous independence hypothesis, contextual information during day $n$ for each base-arm $k_t$ with the context vector $x_{n,t}$ of dimension $d$ is added. The linear assumption gives:

$$\forall n \in [N], \forall t \in [T], \mu_t = x_{n,t} \theta_t. \quad (5)$$

Using Linear Thompson Sampling [13], each base-arm's reward $r_t$ is assumed to be given by a Gaussian distribution
**Algorithm 1 Independent MAB**

**Input:** $\forall t \in [T], \hat{\theta}_t := 0, \sigma_t^2 := 1, \forall k \in \{0, 1\}, N_{k_t}(0) := 0$

1: **for** $n = 1, ..., N$ **do**

2: $E_i(0) := \text{SoC}_i^a \times E_i^{\text{max}}$

3: $\mathbf{s}_o := \emptyset$

4: **for** $t = 1, ..., T$ **do**

5: Sample $\tilde{\theta}_t \sim \mathcal{N}(\hat{\theta}_t, \sigma_t^2)$

6: $\tilde{\mu}[t] := \tilde{\theta}_t$

7: **end for**

8: **for** $t = 1, ..., T$ **do**

9: Compute number of charging instants $m(t)$ to reach $\text{SoC}_i^*$ from Eq. (2)

10: Define set $\mathcal{S}(t) := \{0, 1\}^{T-t+1}$

11: Choose $\mathbf{s}_t := \arg\max_{\mathbf{s} \in \mathcal{S}(t), ||\mathbf{s}||_1 = m(t)} [\mathbf{s}_o, \mathbf{s}]^\top \tilde{\mu}$

12: $\mathbf{s}_o := [\mathbf{s}_o, k_t]$

13: **if** $k_t = 1$ **then**

14: Apply congestion management

15: Observe reward $r_t$ from Eq. (4)

Update base-arm distributions $\mathcal{N}(\hat{\theta}_t, \sigma_t^2)$:

16: $N_{k_t}(t) := N_{k_t}(t-1) + 1$

$$ \hat{\theta}_t := \frac{r_t + N_{k_t}(t-1)\hat{\theta}_t(t-1)}{N_{k_t}(t)} $$

18: $\sigma_t^2 := \frac{1}{N_{k_t}(t)}$

19: **end if**

20: **end for**

21: **end for**

$\mathcal{N}(\mu_t, \sigma^2)$ and the prior for the unknown parameter $\theta_t$ is also Gaussian $\mathcal{N}(\hat{\theta}_t, \sigma^2 A_t^{-1})$. The contextual independent MAB is described by Algorithm 2.

5) **Combinatorial MABs:** Here, the independence assumption is dropped, and it is assumed that the reward of each base arm is linearly correlated with the others. Exploiting this dependence may lead to better performance compared to independent MABs (either context-free or contextual). With $\mathbf{s} \in \mathcal{S}$ the combinatorial vector of dimension $T$ and the unknown parameter $\theta$ of dimension $T$:

$$ \forall \mathbf{s} \in \mathcal{S}, \mu_{\mathbf{s}} = \mathbf{s}^{\top}\theta. \quad (6) $$

Using Linear Thompson Sampling [13], each super-arm's reward $r_{\mathbf{s}}$ is assumed to be given by a Gaussian distribution $\mathcal{N}(\mathbf{s}^{\top}\theta, \sigma^2)$ and the prior for the unknown parameter $\theta$ is also Gaussian $\mathcal{N}(\hat{\theta}, \sigma^2 A^{-1})$. Algorithm 3 presents the pseudocode for the combinatorial Thompson sampling MAB (context-free and contextual). The case where $d=0$ and hence $\mathbf{x}_n = \emptyset$ corresponds to the combinatorial Thompson sampling MAB (i.e. context-free).

6) **Contextual combinatorial MABs:** Contextual information is added in the form of a context vector $\mathbf{x}_n$ of dimension $d$ concatenated with the super-arm vector $\mathbf{s}$, as described below. The contextual combinatorial MAB is described by Algorithm 3.

$$ \forall n \in [N], \forall \mathbf{s} \in \mathcal{S}, \mu_{\mathbf{s}} = [\mathbf{s}, \mathbf{x}_n]^{\top}\theta \quad (7) $$

**Algorithm 2 Contextual Independent MAB**

**Input:** $\forall t \in [T], A_t := I_{d,d}, \mathbf{b}_t := 0_d, \hat{\theta}_t := 0_d, \sigma > 0$

1: **for** $n = 1, ..., N$ **do**

2: $\mathbf{s}_o := \emptyset$

3: $E_i(0) = \text{SoC}_i^a \times E_i^{\text{max}}$

4: **for** $t = 1, ..., T$ **do**

5: observe context $\mathbf{x}_{n,t}$

6: sample $\tilde{\theta}_t \sim \mathcal{N}(\hat{\theta}_t, \sigma^2 A_t^{-1})$

7: compute $\tilde{\mu}_t := \mathbf{x}_{n,t}^{\top}\hat{\theta}_t$

8: $\tilde{\mu}[t] := \tilde{\mu}_t$

9: **end for**

10: **for** $t = 1, ..., T$ **do**

11: Compute number of charging instants $m(t)$ to reach $\text{SoC}_i^*$ from Eq. (2)

12: Define set $\mathcal{S}(t) := \{0, 1\}^{T-t+1}$

13: choose $\mathbf{s}_t := \arg\max_{\mathbf{s} \in \mathcal{S}(t), ||\mathbf{s}||_1 = m(t)} [\mathbf{s}_o, \mathbf{s}]^{\top} \tilde{\mu}$

14: $\mathbf{s}_o := [\mathbf{s}_o, k_t]$

15: **if** $k_t = 1$ **then**

16: Apply congestion management

17: Observe reward $r_t$ from Eq. (4)

Update base-arm distribution $\mathcal{N}(\hat{\theta}_t, \sigma^2 A_t^{-1})$:

18: $A_t := A_t + \mathbf{x}_{n,t}^{\top}\mathbf{x}_{n,t}$

19: $\mathbf{b}_t := \mathbf{b}_t + r_t$

20: $\hat{\theta}_t := \mathbf{b}_t^{\top} A_t^{-1}$

21: **end if**

22: **end for**

23: **end for**

**Algorithm 3 Contextual Combinatorial MAB**

**Input:** $A := I_{T+d,T+d}, \mathbf{b} := 0_{T+d}, \hat{\theta} := 0_{T+d}, \sigma > 0$

1: **for** $n = 1, ..., N$ **do**

2: $r_n := 0$

3: $\mathbf{s}_n := 0_T$

4: Observe context $\mathbf{x}_n$

5: Sample $\tilde{\theta} \sim \mathcal{N}(\hat{\theta}, \sigma^2 A^{-1})$

6: **for** $t = 1, ..., T$ **do**

7: $\mathbf{s}_o := \emptyset$

8: Compute number of charging instants $m(t)$ to reach $\text{SoC}_i^*$ from Eq. (2)

9: Define set $\mathcal{S}(t) := \{0, 1\}^{T-t+1}$

10: choose $\mathbf{s}_t := \arg\max_{\mathbf{s} \in \mathcal{S}(t), ||\mathbf{s}||_1 = m(t)} [\mathbf{s}_o, \mathbf{s}, \mathbf{x}_n]^{\top} \tilde{\theta}$

11: $\mathbf{s}_o := [\mathbf{s}_o, k_t]$

12: **if** $k_t = 1$ **then**

13: Apply congestion management

14: Observe reward $r_t$ from Eq. (4)

15: $r_n = r_n + r_t$

16: **end if**

17: **end for**

Update base-arm and super-arm distribution $\mathcal{N}(\hat{\theta}, \sigma^2 A^{-1})$:

18: $A := A + [\mathbf{s}_o, \mathbf{x}_n]^{\top}[\mathbf{s}_o, \mathbf{x}_n]$

19: $\mathbf{b} := \mathbf{b} + r_n$

20: $\hat{\theta} := \mathbf{b}^{\top} A^{-1}$

21: **end for**
<figure><img src="image_3.png" /><figcaption>Fig. 3. Cumulative average reward over all EVs (for 5 samples and a confidence interval of 95%)</figcaption></figure>

dent bandits show improvements ranging from 9% (with only price context) to 17% (with both price and day type contexts) compared to the baseline. Similarly, combinatorial bandits improve by 26% when context is introduced, compared to their context-free counterparts. This translates to a 14% improvement over the baseline, mitigating the negative effects of modifying Thompson sampling's theoretical setting.

Beside the asymptotic value of the algorithms, convergence speed must also be considered. It is important to recall that in this problem, a cold start is considered (i.e. the bandits do not have any information on the environment, which preserves privacy). Although this may increase convergence time, Fig. 3 shows that all algorithms tend to reach at least 95% of their asymptotic value after approximately 150 days (see Table I). The values reached by the contextual bandits (outperforming the baseline, as mentioned before) already attain at least 85% of their asymptotic value after only 50 days of training, while at least 77.8% of their asymptotic value is reached after just 25 days of training. This rapid convergence tends to decrease the cost of training, i.e. the cost associated with degraded performance during the first phases of training due to the necessary exploration of the algorithm in search for better solutions. This cost applies of course if training is performed in real-life, as opposed to training performed on a simulator. Compared to real-life training, simulator training may require larger computing resources while introducing more uncertainties.

These results should be qualified in regard of the relatively homogeneous nature of agents. Although this study shows performance on a well designed set of agents (sufficient energy supply) with deterministic preferences and heterogeneity with telecommuting days, further studies will enlighten the system behavior with stochastic preferences and a lesser amount of energy, bringing a greater heterogeneity and fairness concerns.

## V. CONCLUSIONS

This paper evaluated 6 bandit-based algorithms for decentralized EV smart charging with mobility and grid constraints. Contextual algorithms outperformed baselines by up to 17%, with rapid convergence reaching 77.8% of asymptotic value

<table><caption>TABLE I
CUMULATED AVERAGE REWARD (% OF VALUE ON DAY 400)</caption><thead><tr><th rowspan="2"></th><th colspan="3">Number of days</th></tr><tr><th>25</th><th>50</th><th>150</th></tr></thead><tbody><tr><td>Baseline</td><td>95.9</td><td>96.9</td><td>99.3</td></tr><tr><td>MILP</td><td>92.2</td><td>97.1</td><td>99.6</td></tr><tr><td>Combinatorial MAB</td><td>93.0</td><td>95.0</td><td>99.6</td></tr><tr><td>Contextual combinatorial MAB (price)</td><td>81.2</td><td>85.8</td><td>95.9</td></tr><tr><td>Contextual combinatorial MAB (price, day type)</td><td>80.6</td><td>85.6</td><td>95.6</td></tr><tr><td>Contextual independent MAB (price)</td><td>79.9</td><td>85.7</td><td>97.4</td></tr><tr><td>Contextual independent MAB (price, day type)</td><td>77.8</td><td>86.6</td><td>97.4</td></tr><tr><td>Independent MAB</td><td>83.7</td><td>85.6</td><td>94.5</td></tr></tbody></table>

within 25 days. The minimum gap to the theoretical optimum from an omniscient, centralized agent is 14%. This decentralized approach enables practical deployment by distributing computation across EV agents, with the most computationally intensive algorithm requiring only 200 µs per EV per timestep. The system maintains privacy by sharing only minimal information—instantaneous power and agent count—for congestion management. Future work will address a fully decentralized version with no private information shared to the aggregator, and explore different aggregator approaches to maximize renewable energy consumption, while investigating greater heterogeneity in EV preferences to better reflect real-world charging scenarios.

## REFERENCES

[1] RTE, "Energy Pathways to 2050 - Key results", Technical report, 2021.

[2] L. Li, W. Chu, J. Langford, R. E. Schapire, "A contextual-bandit approach to personalized news article recommendation", in Proc. 19th international conference on World wide web, 2010.

[3] H. Dakdouk, R. Féraud, N. Varsier, P. Maillé, R. Laroche, "Massive multi-player multi-armed bandits for IoT networks: An application on LoRa networks", Ad Hoc Networks, 2023.

[4] D. Delande, P. Stolf, R. Féraud, J.M. Pierson, A. Bottaro, "Horizontal scaling in cloud using contextual bandits" European Conference on Parallel Processing, 2021.

[5] A. Barbato, L. Chen, F. Martignon, S. Paris, "A Multi-Armed Bandit Formulation for Distributed Appliances Scheduling in Smart Grids", in Proc. IEEE GreenCom conference, Taipei, Taiwan, 2014.

[6] Z. Zhao, A. L. Liu, Y. Chen, "Electricity Demand Response under Real-Time Pricing: A Multi-armed Bandit Game", in Proc. of Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APISPA ASC), Honolulu, USA, 2018.

[7] M. Brégère, "Stochastic bandit algorithms for demand side management", PhD thesis, Université Paris-Saclay, France, 2020.

[8] S. Jain, S. Gujar, "A Multiarmed Bandit Based Incentive Mechanism for a Subset Selection of Customers for Demand Response in Smart Grids", in Proc. of the AAAI Conf. on Artificial Intelligence, 2020.

[9] S. Chandlekar, S. Jain, S. Gujar, "A Novel Demand Response Model and Method for Peak Reduction in Smart Grids", in Proc. IJCAI conference, Macao, China, 2023.

[10] E. Spyrou, V. Kappatos, A. Tromaras, A. Anagnostopoulou, "A Multi-armed Bandit Approach for Electricity Reward of Smart Homes in a Smart Power Grid", 2024.

[11] R. Sutton, A. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

[12] S. Zafar, R. Féraud, A. Blavette, G. Camilleri, H. Ben Ahmed, "Decentralized Smart Charging of Large-Scale EVs using Adaptive Multi-Agent Multi-Armed Bandits", in Proc. CIRED conference, Rome, Italy, 2023.

[13] S. Agrawal, N. Goyal, "Thompson Sampling for Contextual Bandits with Linear Payoffs," in Proc. PMLR conference, 2013, revised in Feb. 2014 (arXiv website)

[14] NREL, Oahu Solar Measurement Grid, dataset, accessed 07/01/2025.