# Versatile and Self-Adapting Smart Home Control

Fabien Sanchez, Vincent Debusschere, Rémy Rigo-Mariani, Antoine Labonne

► To cite this version:

Fabien Sanchez, Vincent Debusschere, Rémy Rigo-Mariani, Antoine Labonne. Versatile and Self-Adapting Smart Home Control. CIRED, Jun 2025, Genève (CH), Switzerland. pp.5, 10.1049/icp.2025.1483. hal-05189359

HAL Id: hal-05189359

https://hal.science/hal-05189359v1

Submitted on 28 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Versatile and Self-Adapting Smart Home Control

Fabien Sanchez*, Vincent Debusschere, Rémy Rigo-Mariani, Antoine Labonne

Univ. Grenoble Alpes, CNRS, Grenoble INP, G2Elab, F-38000 Grenoble, France
* fabien.sanchez@g2elab.grenoble-inp.fr

**Keywords:** DEEP REINFORCEMENT LEARNING, ENERGY MANAGEMENT SYSTEM, BATTERY, PV PANELS

## Abstract

The research focuses on the control of a residential photovoltaic and battery system, designed to mitigate uncertainties from the distribution grid perspective while following prescheduled power profiles at the residential meter level. Specific attention is paid to reduced communication requirements, contrary to the traditional energy management strategy. This autonomy is achieved here using reinforcement learning, which has proven effective in the dynamic management of energy systems. The controller's effectiveness is compared to other methods, such as rule-based controls and optimization approaches. In the use case, the objective is to follow in real-time a power profile based on the moving average of the net load over a week. The controller's decision-making process is driven by a proximal policy optimization agent, which uses sequential temporal inputs such as the target power profile, the state of charge, and the net load data from the previous time step. To enhance the performance of the controller, the research explores the integration of a neural network architecture inspired by image processing techniques.

Performance evaluation criteria include the daily energy deviation and the daily avoidable energy deviation from the target profile.

## 1 Introduction

Distribution grids are facing new operational exigences with the democratization of high-power charging requirements for electric vehicles or the wide integration of distributed photovoltaic (PV) panels. Flows are no longer solely downward, and the legacy grids are increasingly constrained. When focusing on consumption, one way to mitigate those constraints consists of consumers adapting their consumption in response to requests from the distribution system operator (DSO) [1]. The DSO shall also improve prediction capabilities regarding the individual's consumption [2]. The first approach is mainly based on a contractual motivation between the end-user and the DSO, or via dynamic tariffs. However such a demand response (DR) scheme is subject to the risk of loss of communication between the two actors, and issues of fairness between consumer service and the global welfare [3]. The second approach is the prediction of the individual's consumption at different time scales, to mitigate uncertainty. Short-term prediction is typically achieved using solutions based on stochastic [4] or data-driven methods [5]. With the development of smart meters, more data become available which ensures the ascension of data-driven methods [6]. Especially, deep learning solutions and deep reinforcement learning (DRL) provide promising solutions for energy management systems, with the capacity of the artificial neural network (ANN) to handle exogenous data.

In this context, the presented research focuses on a control scheme that could prove resilient to uncertainties and autonomous in the absence of communication (wanted or not). The considered case study consists of a PV-battery energy storage system (PBESS) in a smart home. To mitigate uncertainty,

the end-user must fulfill a commitment to the power exchanges with the upstream distribution grid, defined here in the form of a reference load curve. The commitment schedule is based on his average consumption over the last 7 days, but could be anything else, for instance considering specific grid constraints or other economic incentives. We assumed that the DSO and the customer contractually agree on the schedule that can be updated regularly (e.g daily, weekly, monthly, etc.).

Based on a reinforcement learning (RL) solution, the proposed controller uses temporal data and forecasts to control the battery in real-time. The objective is to keep the net consumption in the scheduled range. The solution's performance is evaluated using the daily squared energy deviation (SED) with the schedule and the avoidable squared energy deviation (ASED). Also, reference optimisation-based and rule-based controllers are introduced for baselining purposes.

The paper is structured as follows: Section 2 presents the study case of the proposed controller, Section 4 presents the test results and Section 5 concludes the research carried out in this contribution.

## 2 Study Case

Fig. 1 depicts a diagram of the studied PV-battery installation.

### 2.1 Hardware Components

The PBESS is composed of five components within a modular approach. The controller sends a command ($p_{n,t}^*$) to the local controller of the battery converter. PV panels are sized to fit a classic installation of a dozen square meters in the south of France. The size of the battery (11.8 kWh) is based
<figure><img src="image_2.png" /><figcaption>Fig. 1. Schematic of the PBESS</figcaption></figure>

on solutions available on the market for residential batteries and the converters are sized according to the observed power flows. The battery power ($p_{n,t}^{bat}$) and the PV production ($p_{n,t}^{pv}$) are tracked by sensors. The SOC of the battery ($SOC_{n,t}$) is given by the SOC estimator to the controller. A smart-meter monitors the power exchange with the grid ($p_{n,t}^{net}$) and the power consumed by the user ($p_{n,t}^{load}$).

## 2.2 Control Strategy

The proposed approach is designed to reduce the end-user's consumption uncertainty for the DSO to better predict grid power profiles. In this study case, a reference profile ($p_{n,t}^{sch}$) is scheduled (and serves as a reference) based on a 7-day moving average, with a fixed margin of flexibility ($\pm f_{n,t}$) that remains constant in time in this paper but can be updated if needed.

# 3 Methodology

## 3.1 Data Source

The data is extracted from the European project Collaborating Smart Solar-powered Micro-grids (CoSSMic) which regroups 11 households in southern Germany*. For our application, the consumption is aggregated, and a profile is generated using the mean net consumption ($p_{n,t}^{net}$) of the last seven days with a resolution of 30 min.

Certifying the representativeness of the training subset, K-means clustering and proportional selection on the training dataset ensures better convergence and stability during the agent training [7]. Here, it was applied on SOC time-series to classify daily SOC patterns with a rule-based controller. Using the Elbow method, three clusters were determined using an Euclidean barycentre calculation. Identified clusters are: i) days with battery depletion (SOC running out), ii) days with excess PV generation (battery fully charged) and iii) middle-range days. The dataset is thus split into sets of 10 days according to the proportional selection method given in [7]. Each set has a representative portion of profiles from each cluster. Three

sets of 10 days are selected to train the agent, and the days are randomly shuffled. Another three different sets of 10 days are dedicated to testing the performance of the references and proposed solutions.

## 3.2 References Controllers

Two references are introduced to assess the performances of the proposed solution.

### 3.2.1 Optimization-Based
The objective of a PBESS is to take advantage of the battery and PV production to follow a daily schedule, regardless of the actual end-user's energy consumption. A first baseline controller can take the form of an optimization problem such as (1) to find the best PBESS controls $P_n^{bat}$ along a day $n$ (the time horizon is 24 hours) to minimize the daily deviation $E_n$ from the scheduled range. The purpose of this baseline is to generate the reference solution, achieved using perfect forecast. The power exchanged by the battery is denoted $p_{n,t}^{bat}$ and, by analogy, the SOC of the PBESS $SOC_{n,t}$, the power exchanged with the grid $p_{n,t}^{netload}$, the power produced by PV panels $p_{n,t}^{pv}$ and the power profile to follow $p_{n,t}^{schedule}$. The time resolution is $\delta t = 30$ min.

$$ \min_{p_{n,t}^{bat}} E_n(P_n^{bat}) = \sum_{t=0}^{T} \Gamma_{n,t} \times ((p_{n,t}^{sch} - p_{n,t}^{net}) \times \delta t)^2 \quad (1a) $$

s.t.

$$ p_{min}^{bat} \leq p_{n,t}^{bat} \leq p_{max}^{bat}, \quad (1b) $$

$$ p_{n,t}^{bat} = p_{n,t}^{-} + p_{n,t}^{+}, \quad (1c) $$

$$ \Gamma_{n,t} = \begin{cases} 0 & \text{if } |p_{n,t}^{sch} + p_{n,t}^{net}| \leq f_{n,t}, \\ 1 & \text{else.} \end{cases}, \quad (1d) $$

$$ p_{n,t}^{net} = p_{n,t}^{pv} + p_{n,t}^{load} + p_{n,t}^{bat}, \quad (1e) $$

$$ SOC_{n,t} = SOC_{n,t-1} + \frac{1}{C} (\frac{p_{n,t}^{+}}{\eta_{+}} - p_{n,t}^{-} \eta_{-}), \quad (1f) $$

$$ SOC_{min} \leq SOC_{n,t} \leq SOC_{max}, \quad (1g) $$

$$ SOC_{n,0} = SOC_{n+1,0} = 50\% \quad (1h) $$

In this optimization problem, $C$ is the capacity of the battery in kWh; $\eta_+$ and $\eta_-$ are the efficient coefficients during the charge ($p_{n,t}^+$) and discharge ($p_{n,t}^-$); $SOC_{min}$ and $SOC_{max}$ are respectively fixed to 17% and 97%; $p_{max}^{bat}$ and $p_{min}^{bat}$ are 5 kW and -5 kW (1b) for the study case. The constraints (1f) to (1h) represent the battery model and the avoidance of out-bounding behavior. The indicative function $\Gamma_{n,t}$ (1d) minimizes the out-of-flexibility-margin power ranges.

### 3.2.2 Rule-Based
A basic rule-based controller (shown in Fig. 2) is implemented to act as a baseline without a perfect forecast. At each time step, it computes the next SOC if it follows perfectly the desired profile and completes the power request if the constraints of the battery allow it.

<footnote>*CoSSMic Project Data available on data.open-power-system-data.org</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 2. Rule-based synoptic</figcaption></figure>

### 3.3 Reinforcement Learning

In essence, DRL is a combination of deep networks and reinforcement training. Q-learning is among the most cited [8], but it cannot handle continuous action spaces and is thus not efficient with continuous observation spaces [9]. Two categories fit the current study case: soft actor critic (SAC) and proximal policy optimization (PPO). Compared to the SAC, the PPO is highly parallelizable and works well in low-cost environments of simulation. PPO is an on-policy algorithm that ensures stable policy updates by employing a clipped objective function [10]. It constrains the policy change within a small range preventing large and destabilizing updates during training. Well-balanced between exploration and exploitation, PPO is perfectly suitable for a continuous reward environment. It is selected to address the presented research.

#### 3.3.1 Action Space
The agent has to control the exchanged power from the battery to the system. It requires continuous power in the range [-5 kW, 5 kW]. A scaling factor $\sigma = 5$ is used, as indicated in (2), to rescale the range of ANN's output ([-1, 1]).

$$a_{n,t}^{scaled} = a_{n,t} * \sigma \quad (2)$$

#### 3.3.2 Observation Space
The ANN receives as input the state vector (3) where $p_{n,t}^*$ is the corresponding command to obtain $p_{n,t}^{sch} - p_{n,t}^{net} = 0$.

$$S_{n,t} = \{SOC_{n,t-1}, p_{n,t-1}^{bat}, t, f_{n,t}, p_{n,t}^*\} \quad (3)$$

Several observation configurations are under study, the first state vector uses (3) and the second (4). This type of sequential input brings interest to other data treatments such as convolutional layers [11].

$$S_{n,t}^{f} = \{SOC_{n,t-1}, p_{n,t-1}^{bat}, t, f_{n,t}, p_{n,t}^{*}, \dots, p_{n,t+6}^{*}\} \quad (4)$$

Each parameter in the state vectors is individually normalized using the min-max method to ensure better convergence

and stability during the training.

$$S_{n,t}[i] = \frac{S_{n,t}[i] - S_{min}[i]}{S_{max}[i] - S_{min}[i]} \quad (5)$$

#### 3.3.3 Reward Function
The control objective is to minimize the gap between the actual grid power and the scheduled range as expressed in (1). By definition, the agent maximizes the reward obtained over each episode. The corresponding reward function consists of two parts. The first one is (6), with (7) giving the objective to the ANN, and the second part, (8), ensures valid actions and penalizes the agent if the action brings the SOC out-of-bounds.

$$r_{n,t} = -\Gamma_{n,t} \times ((p_{n,t}^{sch} - p_{n,t}^{net}) \times \delta t)^2 - p_{n,t}^1 \quad (6)$$

with:

$$\Gamma_{n,t} = \begin{cases} 0 & \text{if } |p_{n,t}^{sch} - p_{n,t}^{net}| \le f_{n,t} \\ 1 & \text{else.} \end{cases} \quad (7)$$

$$p_{n,t}^1 = \begin{cases} -25 & \text{if } SOC_{n,t+1} \notin [SOC_{min}, SOC_{max}] \\ 0 & \text{else.} \end{cases} \quad (8)$$

### 3.4 Agent Configurations

Different agent configurations are compared. The agent consists of an input layer, two dense layers with the same number of neurons, and an output layer which gives an output in the range [-1,1].

#### 3.4.1 PPO without Forecast (PPO)
Uses the last state $S_{n,t}$, expressed in (3), to predict the next command $P_{n,t}^*$. The size of the dense layer is tested within [64, 128, 256].

#### 3.4.2 PPO with Forecast (fPPO)
Identical to PPO, but with the state $S_{n,t}^f$ (4). The objective is to study the effectiveness of considering three hours of forecasted data as additional inputs.

#### 3.4.3 PPO with Partial CNN Extractor (CnnPPO)
Generates commands based on the state $S_{n,t}^f$ (4) but a pre-process is conducted on the input data. A CNN extractor is implemented to catch temporal dependencies over the $S_{n,t}^f$ [4, ..., 9] (4) terms with a kernel size equal to 3, padding to 1, and stride to 1.

## 4 Mains Results

Controllers and simulation environments are implemented using Python on a 10-core laptop with a GPU. The training of the RL agents is done using the Stable Baseline3 library. The agents are trained over 1 million time steps, during 14 min (25 min for the CNN configurations).

### 4.1 Evaluation Methods

To evaluate the performance of the proposed solutions, two metrics are computed over the test set of $D = 30$ days, with a time horizon $T = 24$ hours. The first metric quantifies the
deviation around the scheduled range as shown in (9). Each
deviation incurs a cost penalty.

$$
E = \sum_{n=0}^{D} \sum_{t=0}^{T} \Gamma_{n,t} \times ((p_{n,t}^{\text{schedule}} - p_{n,t}^{\text{netload}}) \times \delta t)^2 \quad (9)
$$

The second metric characterizes the ability of the solu-
tion to anticipate and avoid out-of-range commands. At each
“out-of-bound” behavior, we evaluate if the controller could
have avoided it in the last three hours by acting differently as
explained in Algorithm 1. If it could have, the metric accounts
for the deviation value. If not, the point is not accounted for.

**Algorithm 1 Mean Avoidable Square Energy**

A ← 0

if $p_{n,t}^{\text{schedule}} - p_{n,t}^{\text{netload}} \ge f_{n,t}$ **then**

Compute lacking SOC to be in bounds.

for *i* in range *I* do

▷ Explore last 3 hours

Compute available SOC.

if available SOC enough **then**

$$
A \leftarrow (p_{n,t}^{\text{schedule}} - p_{n,t}^{\text{netload}})^2
$$

Break

else

Reduce the lacking SOC by the available SOC.

end if

end for

if $A = 0$ **then**

The point is unavoidable.

end if

end if

4.2 Results Comparison

Fig. 3 presents the power of the battery obtained with the per-
fect forecasts (best case scenario), the rule-based one, and the
DRL solutions discussed in this paper (PPO64). The sched-
uled range is visible with a constant flexibility factor $f_{n,t}$ equal
to 0.2 kW.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Sample of the power control sequence on one day</figcaption></figure>

The average reward over the last thousand steps is used to
keep the best agent from the training phase. The agent gets

updated every 1440 steps, at the end of the three training sets.
The rewards along with training, shown in Fig. 4, have been
averaged over a sliding window of 1000 steps to improve visi-
bility. Overall, **fPPO64** provides a better average reward during
the training by 5% and converges faster than **PPO64**. The
**CnnPPO64** is the fastest to converge to its maximum reward.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Evolution of the reward during the training</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 5. Performances of proposed solutions</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6 The resiliency of the training with different agents and 10 different seeds</figcaption></figure>

The **PPO64** improves the mean SED by 21.62%, the
**fPPO64** by 15.1% and the **CnnPPO128** by 8.1% compared
to the rule-based solution (Fig. 5). The gap with the optimal solution is reduced by 41.8%, 28.7% and 15.7% respectively. The reduced accuracy of the fPPO64 and CnnPPO128 compared to the PPO64 seems counter-intuitive and suggests that the input enhancement is not worthwhile. It can be explained by the fact that the reward formulation does not take sufficient advantage of prediction. This behavior should not apply to other DRL solutions that have a larger ANN. Agents with more than 64 neurons per layer perform worse on all criteria than PPO64. PPO64 and fPPO64 beat the rule-based on average ASED by 22% and 6% respectively. Other DRL solutions with a larger size achieve worse average ASED by -2.67% to -29.8%.

To assess the reliability of the training and study the stability of the performance on the same test set, each agent is trained 10 times with different seeds (i.e., starting points). The deviation of performance is plotted as an error bar in Fig. 6. fPPO64 is the most stable across the test set but with a lower best performance overall. CnnPPO64 is the most unstable which implies more hyper-parameter fine-tuning.

# 5 Conclusion & Perspectives

A reinforcement learning algorithm (PPO) agent has been successfully implemented to ensure the power balance of a PV-battery energy storage system according to a scheduled profile. The best-presented solution achieves 21.62% better results than a rule-based reference case on the mean squared energy deviation. The extended input with perfect prediction improves training stability and convergence but degrades overall performance. However, the convolutional agent seems to be underperforming compared to the literature elements.

The next step is to study the control in the context of a moving range of flexibility over a day. The controller will be evaluated over a higher time horizon. The DSO can benefit from schedule following for peak shaving, grid participation, and more. These issues will be the subject of further investigation.

# 6 Acknowledgments

This work has been supported by MIAI@Grenoble Alpes, (ANR-19-P3IA-0003).

# 7 References

[1] A. M. Al-Ghaili, Z.-A. B. Ibrahim, A. A. Bakar, H. Kasim, N. M. Al-Hada, B. N. Jørgensen, Z. B. Hassan, M. Othman, R. M. Kasmani, and I. Shayea, "A systematic review on demand response role toward sustainable energy in the smart grids-adopted buildings sector," *IEEE Access*, vol. 11, pp. 64968-65027, 2023.

[2] F. Dewangan, A. Y. Abdelaziz, and M. Biswal, "Load forecasting models in smart grid using smart meter information: a review," *Energies*, vol. 16, no. 3, p. 1404, 2023.

[3] H. T. Haider, O. H. See, and W. Elmenreich, "A review of residential demand response of smart grid," *Renewable and Sustainable Energy Reviews*, vol. 59, pp. 166-178, 2016.

[4] D. W. Van der Meer, J. Widén, and J. Munkhammar, "Review on probabilistic forecasting of photovoltaic power production and electricity consumption," *Renewable and Sustainable Energy Reviews*, vol. 81, pp. 1484-1512, 2018.

[5] S. K. Hora, R. Poongodan, R. P. De Prado, M. Wozniak, and P. B. Divakarachari, "Long short-term memory network-based metaheuristic for effective electric energy consumption prediction," *Applied Sciences*, vol. 11, no. 23, p. 11263, 2021.

[6] R. Lu, S. H. Hong, and M. Yu, "Demand response for home energy management using reinforcement learning and artificial neural network," *IEEE Transactions on Smart Grid*, vol. 10, no. 6, pp. 6629-6639, 2019.

[7] A. Hussain and P. Musilek, "Energy management of buildings with energy storage and solar photovoltaic: A diversity in experience approach for deep reinforcement learning agents," *Energy and AI*, vol. 15, p. 100313, 2024.

[8] B. Jang, M. Kim, G. Harerimana, and J. W. Kim, "Q-learning algorithms: A comprehensive classification and applications," *IEEE access*, vol. 7, pp. 133653-133667, 2019.

[9] F. AlMahamid and K. Grolinger, "Reinforcement learning algorithms: An overview and classification," in *2021 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE)*. IEEE, 2021, pp. 1-7.

[10] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," *arXiv preprint arXiv:1707.06347*, 2017.

[11] M. Aouad, H. Hajj, K. Shaban, R. A. Jabr, and W. El-Hajj, "A cnn-sequence-to-sequence network with attention for residential short-term load forecasting," *Electric Power Systems Research*, vol. 211, p. 108152, 2022.