# Adaptive and Sustainable Smart Environments Using Predictive Reasoning and Context-Aware Reinforcement Learning

Abderrahim Lakehal, Boubakeur Annane, Adel Alti, Philippe Roosevelt, Soliman Aljarboa

► To cite this version:

Abderrahim Lakehal, Boubakeur Annane, Adel Alti, Philippe Roosevelt, Soliman Aljarboa. Adaptive and Sustainable Smart Environments Using Predictive Reasoning and Context-Aware Reinforcement Learning. Future internet, 2026, 18 (1), pp.40. 10.3390/fi18010040. hal-05450367

HAL Id: hal-05450367

https://univ-pau.hal.science/hal-05450367v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

Article

# Adaptive and Sustainable Smart Environments Using Predictive Reasoning and Context-Aware Reinforcement Learning

Abderrahim Lakehal ¹ID, Boubakeur Annane ¹ID, Adel Alti <fnref n="1" />,<fnref n="2" />,*ID, Philippe Roosevelt ³ID and Soliman Aljarboa <fnref n="2" />

## Abstract

Smart environments play a key role in improving user comfort, energy efficiency, and sustainability through intelligent automation. Nevertheless, real-world deployments still face major challenges, including network instability, delayed responsiveness, inconsistent AI decisions, and limited adaptability under dynamic conditions. Many existing approaches lack advanced context-awareness, effective multi-agent coordination, and scalable learning, leading to high computational cost and reduced reliability. To address these limitations, this paper proposes MACxRL, a lightweight Multi-Agent Context-Aware Reinforcement Learning framework for autonomous smart-environment control. The system adopts a three-tier architecture consisting of real-time context acquisition, lightweight prediction, and centralized RL-based decision learning. Local agents act quickly at the edge using rule-based reasoning, while a shared CxRL engine refines actions for global coordination, combining fast responsiveness with continuous adaptive learning. Experiments show that MACxRL reduces energy consumption by 45-60%, converges faster, and achieves more stable performance than standard and deep RL baselines. Future work will explore self-adaptive reward tuning and extend deployment to multi-room environments toward practical real-world realization.

**Keywords:** smart environments; multi-agent systems; context-aware reinforcement learning; pervasive computing; proactive adaptation; energy efficiency; MACxRL

<figure><img src="image_3.png" /></figure>

Check for updates

Academic Editors: Diego Vergara and
Pablo Fernández-Arias

Received: 2 December 2025

Revised: 29 December 2025

Accepted: 6 January 2026

Published: 8 January 2026

**Copyright:** © 2026 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license.

## 1. Introduction

The rapid advancement of embedded systems, the Internet of Things (IoT), and Artificial Intelligence (AI) technologies have enabled the development of smart environments that integrate physical infrastructure with computational intelligence to enhance comfort, efficiency, and sustainability. Smart buildings, homes, and offices exemplify such environments, where interconnected devices, such as lighting, energy monitoring, and security, cooperate to support human activities intelligently and autonomously.

Recent years have witnessed smart connected environments being well-established paradigm for ensuring seamless IoT integration and intelligent context-awareness in order to enable user-friendly environments. Properly capturing and interpreting context is crucial for enabling intelligent systems to make informed decisions. This leads to the concept of

<footnote n="1">¹ Networks and Distributed Systems Laboratory, Computer Science Department, Faculty of Sciences, University Ferhat Abbas Sétif-1, Sétif P.O. Box 19000, Algeria; lakehal.abderrahim@gmail.com (A.L.); baker11@live.fr (B.A.)</footnote>

<footnote n="2">² Department of Management Information Systems, College of Business and Economics, Qassim University, Buraydah 51452, Saudi Arabia; ssjrboa@qu.edu.sa</footnote>

<footnote n="3">³ LIUPPA-T2I/IUT of Bayonne E2S UPPA, University of Pau, 64600 Anglet, France; philippe.roose@iutbayonne.univ-pau.fr</footnote>

<footnote>* Correspondence: a.alti@qu.edu.sa; Tel.: +966-559901163</footnote>
context awareness, where systems use contextual information (e.g., occupancy status, time of day, or user preferences) to adapt their behavior in real time. Context-aware systems are thus capable of providing relevant services, adjusting operational parameters, and optimizing resource usage based on dynamic environments.

With the rapid growth of today's digital-native generation, users are increasingly adopting AI-based assistive technologies. Their dependence on smart devices continues to rise, making daily tasks easier and life more comfortable. As AI continues to advance, smart environments have gained significant prominence by capturing user interest, enabling the automated management of everyday activities with enhanced context-aware capabilities, and responding more effectively to users' evolving needs and preferences through intelligent applications. Building on this advancement, AI techniques, ranging from rule-based systems to machine learning algorithms, have been deployed to interpret sensor data, learn user behavior, and support autonomous decision-making.

Recent years have seen the emergence of various autonomous solutions across smart environments domains, including daily and urgent tasks, operating autonomously with minimal human intervention. Numerous works have proposed intelligent control methods for smart environments [1-25]. They have relied mainly on predefined rules or static decision models, which often lack the flexibility to adapt in complex, evolving environments. These rule-based methods struggle in scenarios with conflicting objectives, such as balancing user comfort with energy efficiency, or when faced with unpredictable contextual changes. Although some efforts have proposed hybrid systems that combine context modeling with decision mechanisms, much of this work relies on human-crafted rules, which limit innovation and generalization. Only few studies leverage Reinforcement Learning (RL) to predict context changes and adjust policies proactively. This represents a significant gap, as RL can automatically learn optimal actions through interaction with the environment, making it well-suited for dynamic and uncertain contexts.

To bridge this gap, we propose MACxRL, a hybrid intelligence model that combines Multi-Agent Systems (MASs) with Context-Aware Reinforcement Learning (CxRL) for scalable and autonomous smart environment control. Distributed agents interpret context, coordinate through shared alerts, and collaborate with a central learning engine to adapt decisions with minimal human intervention. A smart-building case study is modeled in UML and evaluated under different scenarios, demonstrating improved comfort-energy balance, responsiveness, and policy stability. The main contributions of this paper can be summarized as follows:

* First, we design and implement a lightweight short-term energy prediction mechanism based on recent consumption changes and adaptive fallback, which enables fast computation on resource-constrained edge devices.

* Second, we introduce a hybrid decision workflow with coordinated multi-agent control, where local agents produce context-aware predictions, and a centralized CxRL module refines decisions using shared experiences. Agents coordinate via context and energy alerts to prevent conflicting actions across services.

* Third, we conduct a comprehensive comparative study, benchmarking five state-of-the-art smart environment control methods against the proposed design under identical experimental conditions, showing improved convergence, stability, and energy-comfort balance compared to current AI-based control strategies.

The rest of the paper is structured as follows. In Section 2, we introduce a motivating case study to show how our approach can be applied in practice. In Section 3, we briefly discuss related work on smart environments. In Section 4, we model the domain of the smart environment in the UML editor and develop complex rules of the system. Section 5 presents
a reinforcement learning based context-aware multi-agent reasoning model. Section 6 presents verification of the case study, and finally, we conclude in Section 7.

## 2. Motivating Scenario: Intelligent Management of Smart Environment

The following scenario illustrates how our approach operates in practice:

Modern smart buildings integrate systems like lighting, air conditioning (AC), and energy monitoring to enhance comfort and efficiency. Although each subsystem operates autonomously but must collaborate under changing conditions to achieve optimal performance. Their interdependence creates complexity, as decisions in one system affect others.

Figure 1 illustrates a smart office scenario at 8 PM, where two employees are still working. The lights remain at full brightness, and the AC maintains 22 °C. When the energy monitoring system detects consumption reaching 85% of the daily limit, it must reduce load to avoid exceeding targets. Without coordination, lighting may remain bright due to occupancy, and AC may continue cooling based on preset preferences, leading to conflicts between comfort and energy-saving goals. The challenge increases under dynamic conditions, such as people coming and leaving unpredictably, and external temperature changes affecting indoor demand. With decentralized, reactive decisions, each subsystem behaves correctly in isolation but fails to support global coordination.

<figure><img src="image_4.png" /><figcaption>Figure 1. Motivating Scenario: Smart Building Energy-Comfort Management.</figcaption></figure>

As a result, isolated systems often fail to meet global objectives efficiently. A Multi-Agent System (MAS) can solve this by turning subsystems into communicating agents: Agent1 (lighting), Agent2 (AC), Agent3 (energy monitoring), enabling them to observe the environment, exchange messages, and coordinate actions. For instance, if energy consumption exceeds 80%, the energy agent can request the lighting agent to dim lights, transforming isolated units into a coordinated network. Yet, MAS remains insufficient when policies are static. Integrating context-aware reinforcement learning allows agents to
learn from experience and act proactively using time, occupancy, and consumption trends. Instead of reacting only after limits are reached, agents can predict rising load (e.g., nearing 85%), dim lights gradually, or adjust cooling in advance, achieving a smoother balance between comfort and energy.

The goal of this research is to investigate the use of Multi-Agent Context-Aware Reinforcement Learning (MACxRL) to enhance personalization while improving energy efficiency and sustainability in smart environments. By integrating context-aware computing with reinforcement learning techniques, the proposed system aims to learn from user behavior, adapt to dynamic environmental conditions, and autonomously adapt services to balance comfort and energy. The approach seeks to overcome the limitations of traditional rule-based systems through adaptive rule-based reinforcement learning capable of adjusting system behavior in real time. Through coordinated decision-making among distributed agents, the system seeks to reduce unnecessary energy usage, prevent policy violations, and create a more intuitive, efficient, and sustainable smart home experience.

## 3. Related Work

Recent advancements in smart environments have increasingly focused on incorporating intelligent decision-making and adaptive behavior through context-aware systems and multi-agent frameworks. Traditional systems have often relied on rule-based approaches [1,2], which, while effective in static conditions, lack the flexibility to handle dynamic and uncertain environments. These systems typically struggle with conflicting objectives, such as balancing user comfort and energy efficiency, due to their predefined logic and limited ability to learn from real-time data.

To overcome these limitations, researchers have proposed hybrid systems combining context modeling with adaptive logic, where context-aware agents interpret environmental and user data to make informed decisions [3]. However, many of these systems still depend heavily on manually crafted rules, which do not scale well to complex or evolving scenarios.

In recent years, Reinforcement Learning (RL) has been introduced as a promising alternative to static decision logic. RL learns optimal actions through environment interaction, enabling real-time adaptive behavior [4,5]. Despite its potential, the integration of RL into multi-agent and context-aware systems remains relatively limited. The proposed approach addresses this gap by introducing a Multi-Agent Context-Aware Reinforcement Learning (MACxRL) system, which combines the adaptability of RL with the proactivity of context-aware agents. Unlike conventional systems, our architecture allows distributed agents to learn from contextual changes, coordinate proactively, and optimize both local and global objectives dynamically. Similar efforts have been seen in works applying RL to IoT environments [6] and smart grids [7], but few extend this to personalized daily activity management in smart buildings using decentralized, context-driven architectures. By modeling context using a generic UML-based framework and incorporating CxRL agents capable of rule reasoning and prediction, our system enhances adaptability and user autonomy with minimal human intervention, offering a novel, scalable, and intelligent solution for next-generation smart environments.

### 3.1. *Context-Aware Multi-Agent Systems*

Several recent studies have explored the integration of context-awareness into Multi-Agent Systems (MASs) to improve adaptability and intelligence in dynamic environments. In their comprehensive survey, Du et al. [8] highlight several research efforts across various application domains. A key contribution of the survey is the presentation of a unified architecture for Context-Aware Multi-Agent Systems (CA-MASs), which outlines the core components, Sensing, Learning, Reasoning, Prediction, and Action, necessary for intelligent
behavior in agents. This model supports the dynamic acquisition and use of contextual information for proactive decision-making. In the domain of ambient intelligence and smart environments, various CA-MASs have been proposed to personalize user experiences. For example, agents use context such as location, activity, and preferences to control lighting and temperature autonomously, balancing comfort with energy efficiency. The survey also discusses the integration of Large Language Models (LLMs) into MASs, proposing that LLMs can enhance reasoning and language-based communication among agents. This opens new directions for scalable and semantically rich coordination [8]. Despite these advances, challenges remain in context modeling, inter-agent coordination, and scalability. The survey identifies these as critical areas for future research, emphasizing the need for standardization in CA-MAS design.

## 3.2. Multi-Agent Reinforcement Learning (MARL)

The integration of context-awareness into Multi-Agent Reinforcement Learning (MARL) has garnered significant attention, aiming to enhance adaptability and coordination among agents in dynamic environments. Chen et al. [9] introduced a framework handling action and observation delays within multi-agent. By formulating Delay-Aware Markov Games and employing centralized training with decentralized execution, the approach mitigates performance degradation due to delays. However, the reliance on centralized training may limit scalability in large-scale, real-world applications. Chen and Zhang [10] proposed incorporating Bayesian networks into actor-critic methods to model action dependencies among agents, enhancing coordination in cooperative settings. While effective, the approach assumes the availability of accurate context information and may face challenges in environments with partial observability or noisy data. Li and Zhang [11] developed CA-COM, a communication protocol that personalizes messages between agents using attention mechanisms, improving cooperation under bandwidth constraints. Despite its advantages, the method's performance heavily depends on the design of the attention mechanism and may struggle with scalability as the number of agents increases. Iqbal and Sha [12] introduced the AAC framework, which employs attention mechanisms within the critic to focus on relevant parts of the environment, facilitating learning in complex multi-agent settings. However, the model's complexity can lead to increased computational overhead, potentially hindering real-time applications. Foerster et al. [13] addressed the non-stationarity in MARL by proposing methods to stabilize experience replay, such as using importance sampling and fingerprinting. While these techniques improve learning stability, they may not fully resolve non-stationarity issues in highly dynamic environments. A recent study [14] presented a framework that separates message and observation components in agent actions, aiming to clarify the contribution of communication in decision-making. Although this separation aids in understanding agent behaviors, the approach may face difficulties in environments where communication and observations are deeply intertwined.

These studies collectively advance the field of CA-MARL by addressing various challenges such as delay handling, communication efficiency, and learning stability. However, limitations persist, including scalability issues, reliance on accurate context information, and computational complexity. Future research should focus on developing more scalable, robust, and context-sensitive MARL frameworks to better handle the complexities of real-world environments.

## 3.3. *Smart Environments with Context-Aware RL*

The integration of context-aware reinforcement learning into smart environments has garnered significant attention, aiming to enhance adaptability and user-centric automation. In [15], Brenon et al. introduced Arcades, a system leveraging deep reinforcement learning
to adaptively manage voice-controlled smart home devices based on contextual information. The system demonstrated robustness to environmental changes, such as sensor failures. However, its reliance on graphical representations may limit scalability in more complex or heterogeneous environments.

Haji Hosseinloo et al. [16] proposed an event-triggered reinforcement learning approach for controlling building micro-climates, optimizing energy consumption and occupant comfort. By making decisions based on specific events rather than fixed time intervals, the system improved learning efficiency. Nonetheless, the approach may face challenges in environments with frequent or unpredictable events, potentially leading to delayed responses.

Kalivarathan et al. [17] developed INOT, a system enhancing smart home automation through spatial context-aware control, allowing users to issue commands like “turn on the light near the window.” While improving user interaction, the system’s effectiveness heavily depends on accurate spatial modeling and may require substantial initial setup and calibration. Chen et al. [18] introduced CASRL, a meta-learning framework designed to ensure safety in reinforcement learning within non-stationary environments. The approach utilized probabilistic models to adapt to changing contexts. However, the complexity of the model and the need for extensive training data may hinder its applicability in real-time or resource-constrained settings.

Hallou et al. [19] proposed a methodology combining meta-modeling with reinforcement learning to develop context-aware IoT systems, validated through a smart home case study. While the approach provided a structured framework for system development, its generalization to diverse IoT applications and dynamic real-world scenarios remains to be thoroughly evaluated.

Li and Wu [20] developed IConS, an intelligent control system utilizing rule-based reasoning for context-aware automation in smart homes. The system demonstrated efficient performance in predefined scenarios. However, its reliance on static rules may limit adaptability to unforeseen contexts or user behaviors, necessitating manual updates for new situations. In [21], the authors presented an adaptive context caching mechanism employing reinforcement learning to optimize data retrieval in IoT applications. While improving efficiency, the approach's performance is contingent on accurately estimating context lifetimes, which can be challenging in highly dynamic environments.

### 3.4. *Context-Aware and Decision-Making Systems*

Moradi et al. [22] introduced CaaSSET, a framework for model-driven development of Context-as-a-Service (CaaS), enabling reusable context provisioning in IoT systems through meta-models and automated code generation. Unlike their service-oriented design, our approach integrates context acquisition, modeling, and RL-based decision-making into a single adaptive control loop for energy-comfort optimization.

Delnevo et al. [23] proposed an emergency-management tool for smart campuses focused on real-time situational awareness and evacuation support. In contrast, our work targets continuous building control under normal conditions, where context drives multi-agent coordination rather than crisis response.

More recently, Polo-Rodríguez et al. [24] leveraged LLMs and context-aware chatbots using UWB-based localization and activity recognition to enable personalized interaction. Their work focuses on user interaction and assistance, while our system operates at the control layer, selecting actuator actions via reinforcement learning.

Violos et al. [25] provide a survey of context-aware decision systems for sustainability, addressing the need for architectures combining context modeling, cognition, and adaptive
control. Our framework aligns with these recommendations by implementing a multi-agent RL structure with explicit context integration to balance energy and comfort.

### 3.5. Comparative Positioning of the Proposed Work

Table 1 presents a comparison of context-aware reinforcement learning methods and multi-agent approaches in smart environments. Although significant advances have been made, existing approaches still face limitations, including reliance on static rules, high computational cost, limited scalability, and reduced robustness under network delays, sensor uncertainty, and dynamic conditions. Many solutions also assume complete and accurate context information, lacking proactive prediction capabilities, and typically optimize a single objective rather than jointly balancing comfort, energy efficiency, and adaptability.

<table><caption>**Table 1.** Comparison of context-aware RL and multi-agent systems in smart environments.</caption><thead><tr><td>Category</td><td>Related Work</td><td>Key Features and Limitations</td></tr></thead><tbody><tr><td>Communication and coordination in multi-agent RL</td><td>CACOM [11], AAC [12], Message Action Adapter [14]</td><td>(+) Personalized communication via attention<br>(+) Improves cooperation under low bandwidth<br>(-) High computational cost with increasing agents<br>(-) Sensitive to attention mechanism design</td></tr><tr><td>Delay and non-stationarity handling in MARL</td><td>Delay-Aware MARL [9], Stabilizing Experience [13]</td><td>(+) Handles observation/action delays<br>(+) Centralized training with decentralized execution<br>(+) Centralized training limits real-world scalability<br>(-) Stability methods are insufficient in highly dynamic contexts</td></tr><tr><td>Context modeling and rule-based reasoning</td><td>Context-aware Bayesian Actor-Critic [10], IConS [20]</td><td>(+) Probabilistic context modeling via Bayesian networks<br>(+) Rule-based reasoning for smart home control<br>(-) Assumes accurate and complete context info<br>(-) Static rules limit adaptability to new or unseen scenarios</td></tr><tr><td>Safe and adaptive learning in dynamic environments</td><td>CASRL [18], Event-Triggered RL [16]</td><td>(+) Safety-aware meta-learning<br>(+) Trigger-based actions to reduce learning overhead<br>(-) High model complexity and data needs<br>(-) May suffer from delayed actions in unstable events</td></tr><tr><td>Spatial and voice context-aware control</td><td>Arcades [15], INOT [17]</td><td>(+) Voice-driven and spatially aware commands<br>(+) Enhances user interaction and control<br>(-) Graphical models may not scale to complex layouts<br>(-) Spatial modeling setup is resource-intensive</td></tr><tr><td>Meta-modeling and development frameworks</td><td>Context-aware IoT Meta-Modeling [19], Adaptive Context Caching [21]</td><td>(+) Structured framework for developing IoT systems<br>(+) Context caching for efficient IoT data access<br>(-) Limited generalization across domains<br>(-) Depends on the precise estimation of context lifespans</td></tr></tbody></table>

In contrast, MACxRL introduces distributed context-aware agents with lightweight prediction, multi-objective reward optimization, and proactive adaptive behavior. This enables autonomous, scalable, and sustainable smart environments capable of learning from incomplete context, anticipating future states, and continuously balancing comfort and energy efficiency in real-world smart environments.

## 4. Context Modeling of Smart Environment

### 4.1. *Smart Environment Context Model*

In smart environments, *context* refers to any information that describes the situation of users, devices, spaces, or activities, and influences decision-making. Based on recent studies [26], context can be categorized into four main categories: (1) Environmental context, including temperature, illumination level, noise, and energy load; (2) User context, describing occupancy, activity, preferences, and satisfaction levels; (3) System/Device
context, involving actuator states, device status, and network connectivity; and (4) Spatio-temporal context, which relates time of day, location within the building, and routine schedules. To efficiently use this information, a context model is required to structure, represent, and manage context within the system. Several modeling techniques exist: ontology-based, logic-based, markup-based, key-value, and UML-based representation modeling. In this work, we adopt UML modeling due to its clear notation, visual clarity, ease of communication among stakeholders, and suitability for complex system design.

The developed UML graphical model integrates predictive reasoning to support daily activity management in smart environments. The system aims to maintain comfort, safety, energy efficiency, and security, autonomously adjusting environmental conditions such as lighting and temperature, generating energy alerts, and detecting anomalies. Sensors continuously monitor room occupancy, temperature, and energy status, enabling automated actuation decisions such as switching lighting and AC in real-time conditions. The UML model is shown in Figure 2, and agent rules are summarized in Tables 2 and 3.

<figure><img src="image_5.png" /><figcaption>Figure 2. Smart environment UML model and metrics.</figcaption></figure>
<table><caption>**Table 2.** Reactive Rules.</caption><tr><td><b>Agent 1: Lighting Agent</b></td><td>R 1.1: When it is nighttime, and there is at least one person present, the system turns on the light.<br/>∀ t,n (Time(t,"night") ∧ NumberOfPeople(t,n) ∧ (n>0) → TurnOnLight(t))<br/>R 1.2: When no people are present in the room and it is morning, the system turns off the light.<br/>∀ t,n (NumberOfPeople(t,n) ∧ n=0 ∧ Time(t,"morning") → TurnOffLight(t))</td></tr><tr><td><b>Agent 2: Air Conditioning Agent</b></td><td>R 1.1: When it is nighttime, and there is at least one person present, the system turns on the light.<br/>∀ t,n (Time(t,"night") ∧ NumberOfPeople(t,n) ∧ (n>0) → TurnOnLight(t))<br/>R 1.2: When there are no people present in the room and it is morning, the system turns off the light.<br/>∀ t,n (NumberOfPeople(t,n) ∧ n=0 ∧ Time(t,"morning") → TurnOffLight(t))</td></tr><tr><td><b>Agent 3: Air Conditioning Agent</b></td><td>R 3.1: When energy consumption exceeds 80%, the system reduces lights and notifies Ag 1 and 2.<br/>∀ t, e (EnergyConsumption(t,e) ∧ (e>80)→DimLight(t) ∧ Notify(t,Agent1) ∧ Notify(t,Agent2))</td></tr></table>

<table><caption>**Table 3.** Adaptive Rules.</caption><tr><td><b>Agent 1: Lighting Agent</b></td><td>L1.1: Learn optimal dimming levels based on time of day, occupancy patterns, and user feedback.<br/>∀ t,o,f TimeOfDay(t) ∧ Occupancy(o) ∧ UserFeedback(f) → LearnOptimalLevel(t,o,f)<br/>L1.2: Predict future occupancy using past trends and dim lights when occupancy to decrease.<br/>∀ t,p(PastOccupancyTrends(p) ∧ LikelyOccupancyDecrease(t,p)→DimLights(t))</td></tr><tr><td><b>Agent 2: Air Conditioning Agent</b></td><td>L2.1: Learn optimal temperature based on occupancy density, time, and outdoor temperature.<br/>∀x ∀t ∀o ∃s (Occupancy(x) ∧ TimeOfDay(t) ∧ OutdoorTemp(o) → OptimalTempSetpoint(s) ∧ LearnedFrom(x, t, o, s)))<br/>L2.2: Predict upcoming energy and reduce cooling while keeping comfort with acceptable bounds.<br/>∀t (EnergySurgePredicted(t) → (ReduceCooling(t) ∧ MaintainComfort(t)))</td></tr><tr><td><b>Agent 3: Air Conditioning Agent</b></td><td>L3.1: Learn energy usage trends based on time, weather, occupancy patterns, and external events.<br/>∀t ∀w ∀o ∀e ∃u (Time(t) ∧ Weather(w) ∧ OccupancyPattern(o) ∧ ExternalEvent(e) → (EnergyUsageTrend(u) ∧ LearnedFrom(t, w, o, e, u)))<br/>L3.2: Predict future consumption levels and issue early alerts to other agents.<br/>∀c ∀t (Predicted(c, t) ∧ FutureConsumptionLevel(c, t) → ∃a ∀ag (AlertIssued(a, c, t) ∧ ToAgent(ag, a)))</td></tr></table>

## 4.2. Modeling Agents' Rules

To model decision-making behavior, we define two kinds of rules:

* Reactive rules respond to immediate sensor inputs based on predefined conditions.

* Adaptive rules are generated through learning processes and evolve based on user behavior and environmental changes.

We develop complex rules using First Order Logic (FOL) based on UML classes and their attributes. We illustrate this through rules defined for three agents, with adaptive rule learning explained in subsequent sections.

## 4.3. Modeling Agents' Interaction

Unlike conventional smart systems, which rely on centralized logic, the proposed approach supports distributed decision-making and dynamic adaptation. Each agent independently observes the environment, executes actions based on its decision policy,
and receives feedback through reward signals. Through iterative learning, agents improve behavior. In a multi-agent reinforcement learning setting, agents operate within a common environment and may coordinate using shared or independent policies. With continued interactions, the agents converge toward near-optimal policies that maximize cumulative returns.

# 5. MACxRL Approach for Managing Smart Environment

We combine MAS and CxRL to achieve decentralized, adaptive coordination in smart environments. An MAS enables distributed sensing and communication, while CxRL supports learning and context-aware optimization. Together, they enable smart buildings to balance comfort, efficiency, and sustainability under dynamic conditions. CxRL extends traditional RL by incorporating context information, such as time, user preferences, weather, or location. In an MAS, agents observe the environment, interact with other agents, and learn policies over time. For example, an occupied room may require AC in summer but not in winter, illustrating context-aware adaptability.

## 5.1. General Architecture

The system is composed of three main layers: *Perception Layer*, *Back-End Layer*, and *Knowledge Base Layer*. It enables intelligent interaction through multiple autonomous agents, each using a Context-Aware Reinforcement Learning model and coordinated by a global Context Manager. Figure 3 presents the three-tier design.

<figure><img src="image_6.png" /><figcaption>Figure 3. General Architecture of the proposed MACxRL framework.</figcaption></figure>

## 1. Perception Layer

This layer handles environmental sensing and actuation. Sensors monitor various parameters like temperature, light, occupancy, and energy consumption, and actuators perform control actions (e.g., turning on/off lights or air conditioning). It also includes mobile and wearable devices that help users to follow the current context of its smart environment. Sensor communication may be event-driven or periodic (e.g., every 10–30 s), publishing sensed values or alerts when thresholds are crossed.

## 2. Back-End Layer

This is the core processing layer, responsible for context reasoning, learning, and decision-making, divided into several functional modules:
* **MQTT Broker:** serves as a message-passing middleware for real-time communication between the perception layer and the back-end layer.

* **Multi-Agent Context-aware Reinforcement Learning System (MACxRL):** consists of several autonomous agents, embedded with a CxRL model that learn optimal actions over time. Each agent maintains its own knowledge base and operates using two rule types: reactive rules (immediate responses, e.g., *If temperature > 30 °C then activate AC*) and adaptive rules (learned from user behavior, e.g., *gradually adjusting AC settings*). Through interaction with the environment, agents receive reward feedback and refine their policies. MACxRL enables state-aware decision-making, featuring:

- State Awareness perceives dynamic contextual inputs such as time, occupancy patterns, and energy trends.

- Policy Learning learns action policies that balance comfort, energy efficiency, and system constraints.

- *Adaptation* continuously improves decision-making via feedback and experience, enabling proactive rather than reactive behavior.

- **Context Manager:** Acts as a global control and learning orchestrator, handling context coordination, shared updates, rule optimization, and ensuring alignment with user preferences. It updates the rules database with newly learned adaptive rules and ensures that actuator commands are contextually aligned with user preferences and system goals [27]. The detailed functionalities are as follows:

* **Context Collection:** collects data from sensors, user feedback, and environment.

* **Context Modeling:** transforms raw sensor data into structured context. Sensors detect conditions such as occupancy or temperature and enable automated lighting or cooling control. Context is represented using attributes (name, value, state, domain rules), e.g., 25 °C classified as *medium*, with thresholds < 22 °C = *low* and >28 °C = *high*.

* **Policy Optimization:** improves decision-making policies through MACxRL reinforcement learning.

* **Context Prediction and Service Adaptation:** This module predicts future contexts based on lightweight and adaptive time series forecasting and adjusts services proactively. It considers factors such as time of day, temperature, AC status, user activity, and scheduled events to generate continuous predictions. Based on these predictions, the system adapts services by adjusting environmental controls. For example, reducing the temperature before peak consumption, dimming lights before bedtime, or suggesting turning off unused appliances. A dedicated consumption agent monitors energy usage and notifies the temperature agent to act before reaching the energy threshold. This coordination allows the system to balance demand while maintaining comfort. Users receive alerts and recommendations, allowing manual approval or automated response. A feedback loop evaluates prediction effectiveness and user reactions, improving future decisions through learning. This proactive mechanism enhances energy efficiency, comfort, and system responsiveness.

### 3. Knowledge Base Layer

This layer consists of three main databases:

* **Rules DB:** Stores predefined/adaptive rules with metadata such as priority level, rule type (reactive/adaptive), triggered conditions, associated agents, and rule version.

* **User Profile DB:** Maintains personal preferences, historical behavior for personalized, context-aware services, and adaptive decision-making in smart environments. It
includes preferred temperature, lighting levels, humidity, and location preferences (e.g., room-specific configurations). It is defined by user ID, role, and profile version.

* **Context DB:** Maintains context history (time, location, smart home states) to support context awareness and adaptive decisions. It defines three context types: (1) Temporal ( timestamps and event logs), (2) Spatial (location/room ID and occupancy), and (3) Environmental (device states, sensor readings, and user-related data).

## 5.2. System Workflow

As illustrated in Figure 4, the MACxRL workflow follows a cyclic process consisting of: (1) sensing and data collection, (2) context modeling and management, (3) context broadcasting, (4) context prediction, (5) actuation and service adaptation, (6) service deployment, and (7) user interaction and feedback.

<figure><img src="image_7.png" /><figcaption>**Figure 4.** Overview of system workflow steps: sensing, context processing, prediction, learning, actuation, and feedback loop.</figcaption></figure>

### Step 1. Sensing and Data Collection

The workflow begins at the perception layer, where sensors and IoT devices (e.g., motion detectors, wearable devices, smart meters) capture raw environmental and user-related data, such as temperature, occupancy, activity, and device usage. To reduce contextual perception errors, raw sensed values are filtered using moving averages and cross-validated with neighboring sensors when available, reducing noise and perception drift. The collected data is encapsulated as JSON messages and transmitted through an MQTT broker.

An example message is shown below:

```json
{
  "sensorID": "Occ001",
  "type": "occupancy",
  "value": true,
  "timestamp": "2025-05-17T20:00:05Z"
}
```
## Step 2. Context Modeling and Management

Sensor data is forwarded to the Context Collection and Modeling module, where it undergoes filtering, normalization, and format conversion (e.g., JSON/XML). The system maps sensor values to semantic context attributes using domain thresholds. For example, temperature values may be mapped as low/medium/high to support reasoning and decision-making.

## Step 3. Context Data Broadcasting

The Context Manager aggregates all processed contexts, builds a unified context representation, and broadcasts it to all intelligent agents. In parallel, both real-time and historical context are stored in the context database to ensure traceability, trend analysis, and reinforcement learning. An example of the shared context message is shown below:

```json
{
  "room": "room A",
  "timestamp": "2025-05-17T20:00:10",
  "occupancy": true,
  "temp": {"value":22.8,"state":"medium"},
  "energyInst": 0.85,
  "metrics": {"energyConsumption":12.5,"comfortLevel":0.92}
}
```

## Step 4. Context-Aware Prediction and Decision Making (MACxRL)

Once the unified context is broadcast (Step 3) and received by distributed agents, each agent enriches its local state with predicted short-term metrics (Step 4.3), and transmits the updated state to the CxRL Engine. Decision-making then proceeds in two coordinated steps. First, the local agent MDP representation (Step 4.1) enables each agent to observe context, maintain local goals, and exchange coordination messages. Second, CxRL optimization (Step 4.2) trains and updates the global policy, which is then broadcast back to all agents to ensure coherent and synchronized decisions across the environment. This mechanism enables real-time responsiveness while progressively improving adaptability and predictive behavior through continuous CxRL feedback.

### Step 4.1. Reactive and Adaptive Agent State Representation (Local MDP)

Each agent models its decision process as a Markov Decision Process (MDP):

$$MDP = (S, A, P, R, \gamma) \qquad (1)$$

where

* S: State space, including occupancy (yes/no), time, temperature, energy consumption, predicted energy, message from neighboring agents, and last actuator settings.

* A: Action space, e.g., AC settings (22-26 °C), or turn off AC, dimming lights.

* P(s'|s, a): Transition probability between states based on performed actions.

* R(s, a): Reward function, which returns positive values when comfort and energy constraints are satisfied, negative values otherwise.

* γ: Discount factor that balances immediate and future rewards (0 ≤ γ < 1).

Figure 5 presents an example of the light agent MDP and its state transitions.

At runtime, each agent continuously constructs its local state and evaluates the environment. Rather than making decisions independently, the agent shares these observations
with the global CxRL Engine. The reward function is normalized to balance energy efficiency, comfort, and user satisfaction:

$$r = R(s, a, s') = -w_E \frac{E_t - E_{min}}{E_{max} - E_{min}} + w_c \frac{C_t - C_{min}}{C_{max} - C_{min}} + w_s \frac{S_t - S_{max}}{S_{max} - S_{min}} \quad (2)$$

where $E_t$ is energy consumption at time $t$, $C_t$ is comfort level, $S_t$ is user satisfaction score. Comfort level represents how close indoor conditions match preferred user ranges (e.g., temperature, lighting), estimated using sensor values compared against acceptable thresholds. User satisfaction reflects perceived service quality, inferred from interaction feedback, override frequency, and stability of comfort conditions. Higher satisfaction occurs when fewer manual adjustments are needed.

<figure><img src="image_8.png" /><figcaption>Figure 5. MDP model for Light agent.</figcaption></figure>

The weights $w_E$, $w_c$, $w_s \ge 0$ are tunable and satisfy $w_E + w_c + w_s = 1$. Min-max normalization maps all metrics to the range [0, 1], ensuring equal contribution across reward terms. High energy consumption is penalized, whereas enhanced comfort and satisfaction improvements yield positive rewards, enabling balanced, human-aware decision-making. Agents run the MDP loop continuously, applying fast rule-based reactive actions, while long-term adaptive decisions are optimized by the global CxRL model.

### Step 4.2. CxRL Learning and Multi-Agent Coordination

In MACxRL, all agents share their experience with a global reinforcement learning engine. Each agent observes the environment, collects experiences tuples (*state*, *action*, *reward*, *next_state*), which are transmitted to the global CxRL engine. The CxRL engine selects optimal actions using ε-greedy exploration, then sends the selected action back to the corresponding agents for execution. The Q-table is updated centrally using aggregated experience from all agents, enabling faster convergence, cross-agent knowledge sharing, and better generalization of user behavior patterns. Since all agents operate under a unified global policy, multi-agent coordination emerges naturally. For example, IF energy usage exceeds 80%, the Energy Agent broadcasts a coordination message to other agents to trigger power-saving actions such as dimming lights or adjusting AC temperature:

```json
{
  "room": "room A",
  "from": "EnergyAgent",
  "cmd": "dim_AC",
  "reason": "energyInst > 0.8",
  "timestamp": "2025-05-17T20:00:10"
}
```

Upon receiving this event, the lighting agent reacts by adjusting brightness:
```json
{
    "room": "room A",
    "brightness":75,
    "timestamp": "2025-05-17T20:00:11Z"
}
```

Such message cascades ensure coherent behavioral adjustments across agents, such as lowering cooling levels or dimming lights, to avoid energy peaks while preserving user comfort. Although learning is centralized, each agent still models its decision-making using an MDP and refines its strategy following Q-learning principles to avoid conflicts and support global optimization.

The policy that maps states to actions is defined as:

$$ \pi(a | s) = \operatorname*{argmax}_a Q(s, a) \qquad (3) $$

Q-values are iteratively updated according to the Bellman equation:

$$ Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma \max_{a'} Q(s', a') - Q(s, a)] \qquad (4) $$

where $\alpha$ is the learning rate ($0 \le \alpha \le 1$) which controls updates, $\gamma$ is the discount factor ($0 \le \gamma < 1$) that balances immediate and future rewards, $r$ is the reward received after performing action $a$ in state $s$, and $\max_{a'} Q(s', a')$ estimates future return from the next state ($s'$). Over time, continuous agent experience populates the global Q-table, progressively enhancing the shared policy. Consequently, local decisions evolve from rule-driven responses to self-optimizing behavior guided by learned strategies. Algorithm 1 summarizes the centralized Q-learning workflow.

<caption>**Algorithm 1: Q-Learning Process**</caption>

```c
# Initialize Q-table
Q = {}

# Learning loop
for each episode:
    state = get_current_state ();
    action = choose_action (Q, state); # epsilon-greedy strategy
    new_state, reward = take_action (action)

# Update Q-value
Q state, action] = Q state, action] + alpha *
    (reward + gamma * max (Q new_state, a]) - Q state, action])
```

### Step 4.3. Lightweight Context Prediction

Before the global RL model selects an action, each agent predicts short-term energy consumption using a lightweight context prediction mechanism. This mechanism is designed for real-time processing on resource-limited devices such as Raspberry Pi, ensuring low computational overhead and minimal latency.

As shown in Algorithm 2, the prediction module receives as inputs the history of energy usage, the number of past readings $N$, the prediction horizon $H$, and the sampling interval $\Delta t$. It computes the difference between each consecutive reading and estimates an average trend that reflects expected consumption change under stable conditions. To improve robustness against sudden variations, an adaptive mechanism compares the latest energy consumption changes with the historical average. If a significant deviation is detected, trend extrapolation is temporarily suspended, and the last observed value
is retained. Otherwise, standard extrapolation is applied. This strategy prevents error propagation during transient shift while preserving algorithm simplicity. Predicted values are then normalized to the range [0%,100%) to ensure realistic output. Unlike logic control, the prediction module relies only on potential historical records rather than the full historical values to ensure fast and efficient prediction. The prediction module does not trigger actions directly. Instead, its output is appended to the agent state and forwarded to the centralized CxRL engine, where reward feedback gradually adjusts prediction accuracy over time. This integration enhances proactive decision-making, enabling early responses to upcoming energy peaks. Besides, the prediction module is flexible and can be replaced with more advanced predictors (e.g., LSTM, Transformer) that leverage potential historical records for forecasting, without requiring changes to the overall system architecture.

**Algorithm 2: Context Prediction**

**Inputs:**
* windowSize *N*;
* horizon *H*;
* samplingInterval *Δt*;
* energyHistory []: Last *N* energy values;
* threshold *τ*: sudden-change detection threshold

**Outputs:** predictedEnergy;

**Begin**
1. **If** energyHistory.size < *N* **Then**
2.   **return** energyHistory;
3. **for** *i* = 1 to *N* − 1 **do**
4.     *deltas[i−1]* = energyHistory[i] − energyHistory[i−1];
5.     *avgDelta* = *mean*(*deltas*);
6.     *recentDelta* = energyHistory[N−1] − energyHistory[N−2];
7.     *steps* = *H*/*Δt*;
8.     **If** |*recentDelta* − *avgDelta*| > *τ* **Then** //sudden change detected
9.       *predictedEnergy* = energyHistory[N−1] Else
10.      **Else** *predictedEnergy* = energyHistory[N−1] + *avgDelta* * steps;
11.      **Check** *predictedEnergy* to [0%, 100%]
12.    **return** *predictedEnergy*;
**End**

**Step 5. Actuation and Service Adaptation**

After the CxRL engine selects optimal actions, the agent performs a service adaptation stage to refine RL-recommended actions based on contextual information, peer coordination messages, and local constraints. The service adaptation module receives as inputs the current context (ctx), the trained Q-learning model, messages from peer agents, and a set of adaptation rules (e.g., "energy does not exceed 85%."). The context is converted into a suitable numerical feature vector and evaluated by the CxRL model to produce a candidate action, which is mapped to an executable command (e.g., set_temp_24, dim Light_70%, turn_off_AC). A service-adaptation filter then adjusts this command when needed: (1) If a coordination message requests power reduction, then decrease lighting/cooling, (2) If a peer agent issues a constraint, then update the command accordingly, and (3) If prediction/policy detects high energy consumption, then replace/scale the action to maintain constraints. The adaptation strategy is shown in Algorithm 3.
## Step 6. Service Deployment

Once the adapted action is validated, it is dispatched to the respective actuator to update the environment. The new system state (e.g., new temperature, energy load, comfort level) is then recorded and forwarded to the next control cycle.

## Step 7. User Interaction and Feedback Loop

After deployment, the system logs the executed action, resulting state, prediction values, reward, and corresponding context. User feedback, when available, is also captured. This information feeds back into Step 4, enabling continuous learning, real-time policy refinement, and improvement of system responsiveness and user experience over time.

<caption>**Algorithm 3: Actuation and Service Adaptation**</caption>

**Inputs:**
Context *ctx*;
Trained Q-Learning;
coordinationMsgs: queue of incoming messages from other agents.
adaptationRules: optional local rules (e.g., "never energy exceeds 85%")

**Outputs:** appliedAction;

**Begin**
1. *s* = vectorize(*ctx*); # *State vector for RL*
2. *a* = π*predict(*s*); # *RL policy*
3. *cmd* = map *action*a; # *transfer RL action to a concrete service command*
4. **For each** *msg* in coordinationMsgs **do**
5.   **if** *msg.cmd* == "Light_reduction" **then**
6.     *cmd* = reduceBrightness(*cmd*);
7.     **else if** *msg.cmd* == "Light_off" **then**
8.       *cmd* = turnOffLights();
9.   **Apply adaptation rules** (e.g., *energy* < 85%)
10.   **Publish command** to actuator
11.   **Record the current state** for learning and update the action
**End**

# 6. Implementation and Experiments

## 6.1. *Implementation*

The proposed system, MACxRL, is developed using a hybrid architecture consisting of physical IoT sensing devices, edge computing nodes, and cloud intelligence layer. Figure 6 presents the physical deployment architecture of MACxRL, showing how sensing devices, the edge gateway, and the learning back-end interact. IoT sensors collect occupancy, temperature, and energy data and transmit it via Zigbee and Wi-Fi to a Raspberry Pi gateway running MQTT and local processing services. The gateway forwards context information to the reinforcement-learning back-end, which performs context analysis and policy decisions. Control commands are then sent back to smart actuators to adjust lighting and HVAC settings. A user interface provides real-time monitoring and manual control, ensuring efficient sensing-to-actuation flow and low-latency system response.
<figure><img src="image_9.png" /><figcaption>**Figure 6.** Physical Network architecture, including IoT sensors, gateways, and CxRL engine.</figcaption></figure>

## 6.2. Settings

To assess the performance and adaptability of the proposed MACxRL framework, we conduct experiments on five representative baseline strategies: no-coord approach [28] and rule-based controller [29], modern Deep RL methods, namely Deep Q-Networks (DQN) [30] and an Actor-Critic (AC) approach [31], and the LSTM-based predictive controller [32]. These comparison models span a diverse range of control architectures and deployment settings, including non-adaptive deterministic control, manual heuristic control, and non-coordinated actuation policies. This selection enables robust evaluation across both simple fixed-rule environments and more dynamic context-driven scenarios as well as varying capabilities in decision reasoning, responsiveness, and scalability.

All simulations were performed in Python and SecNetworkCloudSim [33], using a custom smart-room environment developed to emulate realistic dynamic power usage, thermal characteristics, lighting behavior, and human comfort preferences. The environment captures stochastic occupancy, thermal drift, HVAC response curves, lighting adequacy, and energy-budget constraints, providing a realistic and controllable platform for multi-agent reinforcement learning. The main evaluation parameters and their corresponding values are summarized in Table 4.

<table><caption>**Table 4.** Parameter settings.</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Q-Learning</td><td>γ = 0.90</td></tr><tr><td>Value Iteration</td><td>α is adaptive</td></tr><tr><td>State Space</td><td>6 states</td></tr><tr><td>Action Space</td><td>75 actions</td></tr><tr><td>Reward Weights</td><td>w<sub>E</sub> = 0.55, w<sub>C</sub> = 0.30, w<sub>S</sub> = 0.15</td></tr><tr><td>Simulation Steps</td><td>200</td></tr><tr><td>Energy Threshold</td><td>80% budget</td></tr><tr><td>User Satisfaction Weight</td><td>w<sub>US</sub> = 0.15</td></tr><tr><td>Reward Stability Weight</td><td>w<sub>RS</sub> = 0.85</td></tr><tr><td>Thermal comfort Ideal band</td><td>24–26 °C</td></tr><tr><td>Comfort Temperature Limits</td><td>22–28 °C</td></tr><tr><td>Thermal Comfort Weight</td><td>w<sub>T</sub> = 0.6</td></tr><tr><td>Lighting Comfort Weight</td><td>w<sub>L</sub> = 0.4</td></tr></tbody></table>

The experiments were conducted across 200, 500, and 1000 simulation steps, and repeated over 10 independent random seeds. This setup allows evaluation of both short-
term responsiveness and long-term policy stability. All reported results are expressed as mean and standard deviation.

## 6.3. Metrics

To comprehensively evaluate the performance of the proposed MACxRL control approach, we adopt three complementary metrics: Energy Consumption (EC), Thermal-Lighting Comfort (TC), and User Satisfaction and Policy Reward Stability (US/PRS). These metrics are designed to jointly assess the effectiveness, reliability, and efficiency of the context-aware sustainable framework.

### (1) Energy Consumption (EC)

EC measures the normalized amount of energy used by lighting and HVAC actions that successfully bring the environment to the desired operating state. A lower EC indicates higher energy efficiency and better sustainable behavior of the system. It is formally defined as:

$$EC = \frac{1}{N} \sum_{t=1}^{N} \frac{E_t}{E_{\text{budget}}} \qquad (5)$$

where $E_t$ is the instantaneous energy consumed at step $t$, $E_{\text{budget}}$ is the predefined 80% energy-budget threshold, and $N$ is the total number of simulation steps.

### (2) Thermal-Lighting Comfort (TC)

TC quantifies the combined thermal and illumination comfort levels experienced by the occupants, normalized between [0, 1]. Maintaining a high TC is crucial to avoid user discomfort, temperature deviation, insufficient lighting, and to ensure a pleasant smart-environment experience. It is given by:

$$TC = \frac{1}{N} \sum_{t=1}^{N} [w_{T}C_{T}(t) + w_{L}C_{L}(t)] \qquad (6)$$

where $C_T(t)$ is thermal comfort at step $t$, $C_L(t)$ is lighting comfort at step $t$, and $w_T$, $w_L$ are weighting factors ensuring balanced contribution.

$$C_T(t) = 1 - \frac{T(t) - T_{\text{ideal}}}{T_{\text{max}} - T_{\text{min}}} \qquad (7)$$

### (3) User Satisfaction and Policy Reward Stability (US/PRS)

US/PRS reflects users' overall willingness to accept system behavior, as well as the robustness and consistency of the learned policy. Since MACxRL adopts a reinforcement-learning strategy that iteratively maximizes rewards, US/PRS serves as an indicator of policy-convergence quality. A high US/PRS implies smoother learning, reduced oscillations, and better long-term user experience. It is computed as:

$$US/PRS = \frac{1}{N} \sum_{t=1}^{N} [w_{US}S(t) + w_{RS}R(t)] \qquad (8)$$

where $S(t)$ is user satisfaction at step $t$, $R(t)$ is reward at step $t$, and $w_{US}$, $w_{RS}$ are weighting factors ensuring balanced contribution.

These metrics provide a holistic evaluation of MACxRL: EC reflects its energy-efficiency performance, TC captures its ability to preserve thermal and lighting comfort, and US/PRS measures the stability, reliability, and user-centric quality of the control process.
## 6.4. Global Decision Making Model Using MDP

We conduct comprehensive experiments using a global MDP state-transition model. Figure 7 illustrates how the global MDP operates as the central control mechanism, optimizing energy consumption while preserving user comfort and satisfaction, thereby validating the method's robustness against diverse environmental conditions. Each state in the global MDP represents the environmental context, including time-of-day (Night), occupancy level (e.g., 2 persons), energy status (Low/Medium/High), and the energy-manager signal (Yes/No). Transitions are annotated with joint actions for lighting (A1), HVAC (A2), and energy management (A3), as well as the corresponding transition probability (P) and reward (R). These labels capture the expected changes in comfort, energy efficiency, and user satisfaction derived from the simulation environment. We select a representative set of state transitions and action combinations to construct the experimental set and conduct all experiments under the same controlled simulation conditions to ensure comparability and stability of the results.

<figure><img src="image_10.png" /><figcaption>**Figure 7.** Global Decision MDP Model.</figcaption></figure>

## 6.5. Energy Consumption Analysis

We evaluate the energy consumption of the smart-environment system under the control of MACxRL, with particular emphasis on achieving energy efficiency without compromising occupant comfort or normal system operation. Using the experimental setup described in Section 2, we systematically compare MACxRL with classical control strategies, no-coord approach [28], and the rule-based controller [29], as well as modern learning-based methods, including Deep Q-Networks (DQN) [30], the Actor-Critic approach [31], and the LSTM-based predictive controller [32].

The results reported in Table 5 show that MACxRL consistently achieves the lowest normalized energy consumption (EC = 0.526) across extended horizons. While DQN [30] and Actor-Critic [31] improve energy efficiency compared to classical controllers by learning adaptive policies, their performance remains affected by higher variance and slower convergence in large action spaces. Similarly, the LSTM-based approach [32] benefits from modeling temporal energy-consumption trends but lacks direct policy optimization and coordinated control, which limits its long-term effectiveness.
<table><caption>**Table 5.** Average Energy Consumption Comparison of MACxRL and Baseline approaches.</caption><thead><tr><th>Simulation Steps</th><th>Model</th><th>Total Energy (Wh)</th><th>Normal. EC</th><th>Reduction</th></tr></thead><tbody><tr><td rowspan="6">200</td><td>No-Coord [28]</td><td>5.87</td><td>0.964</td><td>-</td></tr><tr><td>Rule-Based [29]</td><td>7.42</td><td>0.762</td><td>30.3%</td></tr><tr><td>DQN [30]</td><td>3.64</td><td>0.598</td><td>20.9%</td></tr><tr><td>Actor-Critic [31]</td><td>3.47</td><td>0.571</td><td>38.0%</td></tr><tr><td>LSTM [32]</td><td>3.56</td><td>0.586</td><td>39.4%</td></tr><tr><td>Proposed MACxRL</td><td>4.21</td><td>0.526</td><td>45.5%</td></tr><tr><td rowspan="6">500</td><td>No-Coord [28]</td><td>14.90</td><td>0.526</td><td>-</td></tr><tr><td>Rule-Based [29]</td><td>11.70</td><td>0.748</td><td>21.5%</td></tr><tr><td>DQN [30]</td><td>9.17</td><td>0.585</td><td>38.4%</td></tr><tr><td>Actor-Critic [31]</td><td>8.73</td><td>0.558</td><td>41.4%</td></tr><tr><td>LSTM [32]</td><td>8.95</td><td>0.572</td><td>39.9%</td></tr><tr><td>Proposed MACxRL</td><td>8.00</td><td>0.512</td><td>46.3%</td></tr><tr><td rowspan="6">1000</td><td>No-Coord [28]</td><td>29.50</td><td>0.945</td><td>-</td></tr><tr><td>Rule-Based [29]</td><td>23.15</td><td>0.741</td><td>21.5%</td></tr><tr><td>DQN [30]</td><td>23.15</td><td>0.573</td><td>39.3%</td></tr><tr><td>Actor-Critic [31]</td><td>17.17</td><td>0.549</td><td>41.8%</td></tr><tr><td>LSTM [32]</td><td>17.55</td><td>0.560</td><td>40.5%</td></tr><tr><td>Proposed MACxRL</td><td>15.80</td><td>0.505</td><td>46.4%</td></tr></tbody></table>

In contrast, MACxRL explicitly incorporates context-aware and coordinated decision-making through a global MDP formulation, enabling joint optimization of energy-management actions. This coordination allows MACxRL to proactively balance energy savings and comfort preservation rather than reacting passively to predictions, as in LSTM-based control [32], or learning unstable action-reward mappings, as observed in DQN [30] and Actor-Critic methods [31]. However, MACxRL exhibits low variance and stable convergence over extended simulation horizons, indicating reliable long-term behavior. Complementary comfort and user-satisfaction metrics further confirm that these energy savings are achieved without degrading indoor environmental quality, underscoring MACxRL's suitability for real-world, scalable, and sustainable smart-environment deployments.

## 6.6. *Thermal Lighting Comfort Analysis*

The evaluation metric TC quantifies the mean thermal lighting comfort level required to maintain acceptable indoor environmental quality, serving as a critical indicator of occupant well-being, comfort stability, and long-term usability. This metric is important in practical scenarios where occupant comfort must be preserved alongside aggressive energy-saving strategies. Higher TC signifies greater efficiency, rendering the control mechanism more suitable for deployment in smart environments.

Table 6 reports TC results for all evaluated approaches across extended horizons of 200, 500, and 10,000 simulation steps, enabling assessment of both short responsiveness and long-term stability. Compared with the baseline methods, including the no-coord approach [23] and the rule-based controller [24], the proposed MACxRL method achieves an average TC of 0.84–0.88 across all evaluated models, demonstrating superior comfort-preservation efficiency while maintaining stable actuation performance. In contrast, control techniques
such as the no-coord baseline, which require independent subsystem decisions without contextual awareness, incur substantially larger comfort deviations when environmental conditions fluctuate. Similarly, the rule-based controller achieves the lowest TC values (e.g., 0.67 at 200 steps), reflecting its reliance on fixed heuristics that cannot adapt to dynamic environmental changes.

<table><caption>**Table 6.** Average Lighting Comfort Comparison of MACxRL and Baseline approaches.</caption><thead><tr><th>Model</th><th>200 Steps</th><th>500 Steps</th><th>1000 Steps</th></tr></thead><tbody><tr><td>No-Coord [28]</td><td>0.59 ± 0.11</td><td>0.61 ± 0.10</td><td>0.64 ± 0.08</td></tr><tr><td>Rule-Based [29]</td><td>0.67 ± 0.05</td><td>0.69 ± 0.05</td><td>0.71 ± 0.04</td></tr><tr><td>DQN [30]</td><td>0.67 ± 0.05</td><td>0.80 ± 0.05</td><td>0.71 ± 0.04</td></tr><tr><td>Actor-Critic [31]</td><td>0.78 ± 0.06</td><td>0.82 ± 0.04</td><td>0.83 ± 0.03</td></tr><tr><td>LSTM [32]</td><td>0.78 ± 0.06</td><td>0.82 ± 0.04</td><td>0.84 ± 0.03</td></tr><tr><td>Proposed MACxRL</td><td>0.84 ± 0.01</td><td>0.85 ± 0.01</td><td>0.88 ± 0.01</td></tr></tbody></table>

Advanced learning-based approaches, including DQN [25], Actor-Critic [25], and LSTM-based control [27], improve comfort preservation compared to classical baselines by leveraging learned representations and temporal patterns. However, these methods still exhibit higher variance and slower stabilization, particularly over shorter horizons, due to exploration dynamics (DQN, Actor-Critic) or prediction error accumulation (LSTM). In contrast, MACxRL maintains consistently high TC values with minimal variance across all horizons, highlighting the benefit of explicit coordination and context-aware decision-making.

Overall, these results indicate that MACxRL achieves a favorable balance between comfort preservation, energy efficiency, and control robustness, confirming its practical viability and scalability for sustainable smart-environment deployment.

## 6.7. User Satisfaction and Policy Reward Stability Analysis

To evaluate the user satisfaction and policy-reward stability of MACxRL, we conduct extensive simulations across five representative control strategies, including the no-coord approach [28], and the rule-based controller [29], as well as modern learning-based methods, including Deep Q-Networks (DQN) [30], the Actor-Critic approach [31], and the LSTM-based predictive controller [32]. Table 7 reports the average User Satisfaction and Policy Reward Stability (US/PRS) under varying levels of system adaptiveness. Overall, we observe a consistent improvement in MACxRL's ability to maintain smooth, stable, and comfort-aware control decisions over time. This behavior is important in smart-environment applications, where frequent abrupt control actions can negatively affect user experience and system acceptance.

<table><caption>**Table 7.** Average User Satisfaction Comparison of MACxRL and Baseline approaches.</caption><thead><tr><th>Model</th><th>200 Steps</th><th>500 Steps</th><th>1000 Steps</th></tr></thead><tbody><tr><td>No-Coord [28]</td><td>0.41 ± 0.10</td><td>0.44 ± 0.09</td><td>0.48 ± 0.08</td></tr><tr><td>Rule-Based [29]</td><td>0.63 ± 0.07</td><td>0.65 ± 0.06</td><td>0.67 ± 0.05</td></tr><tr><td>DQN [30]</td><td>0.74 ± 0.06</td><td>0.76 ± 0.05</td><td>0.79 ± 0.04</td></tr><tr><td>Actor-Critic [31]</td><td>0.77 ± 0.05</td><td>0.79 ± 0.04</td><td>0.82 ± 0.03</td></tr><tr><td>LSTM [32]</td><td>0.72 ± 0.06</td><td>0.75 ± 0.05</td><td>0.80 ± 0.04</td></tr><tr><td>Proposed MACxRL</td><td><strong>0.82 ± 0.01</strong></td><td><strong>0.84 ± 0.01</strong></td><td><strong>0.88 ± 0.01</strong></td></tr></tbody></table>

The sensitivity to US/PRS also varies across the compared methods. For instance, MACxRL exhibits substantial gains over the no-coord approach [28], with US/PRS improving from 0.41 to 0.82, representing a major increase in stability and user comfort. In
contrast, the rule-based controller [29] achieves its optimal US/PRS of 0.63 already at moderate adaptiveness, indicating that fixed-rule systems may require fewer adjustments but lack long-term scalability and adaptability. While DQN and Actor-Critic methods [28,31] improve satisfaction relative to classical baselines by learning adaptive behaviors, their performance is affected by higher variance and slower convergence, under non-stationary environmental dynamics. LSTM-based approach [32] leverages temporal prediction of system states, attains competitive US/PRS values but lacks explicit policy optimization and coordination, leading to reduced stability over extended simulation steps. In contrast, MACxRL maintains near-optimal US/PRS with low variance and minimal control overhead, demonstrating that explicit coordination and context-aware reinforcement learning are key enablers of robust, scalable, and user-centric smart-environment control.

## 6.8. *Analysis of Energy-Comfort and User Experience*

Figure 8 presents the average energy consumption, comfort level, and user satisfaction achieved by each control method, offering a clear comparison of their overall performance and operational stability. The proposed MACxRL framework consistently delivers the best results, achieving the lowest energy consumption and the highest comfort and satisfaction scores. It maintains stable performance even under dynamically changing environmental contexts, demonstrating robustness and balance across evaluated metrics.

<figure><img src="image_11.png" /><figcaption>**Figure 8.** Comparison of Energy-Comfort and User Experience of Different Control Models.</figcaption></figure>

The rule-based controller [28] ranks second, achieving moderate energy efficiency and comfort improvements through predefined handcrafted rules; however, its performance degrades when facing rapid or unpredictable context variations. In contrast, the no-coord baseline [29] shows the weakest performance, consuming the highest energy and providing lower comfort and satisfaction levels. This highlights the limitation of static, non-coordinated control mechanisms in adapting to environmental perturbations and managing smart-room dynamics effectively. When compared with advanced deep learning approaches, including deep Q-Networks (DQN) [30], Actor-Critic [31], and LSTM-based control, MACxRL maintains a clear advantage in terms of stability and overall balance [32]. While deep learning models improve energy efficiency and comfort relative to classical baselines by learning adaptive patterns, they exhibit higher variance and sensitivity to non-stationary dynamics, particularly during learning and under abrupt context changes. The LSTM-based approach benefits from temporal trend modeling but lacks explicit policy optimization and coordination, limiting its effectiveness in jointly managing energy, comfort, and user experience. Similarly, DQN and Actor-Critic methods suffer from exploration-induced oscillations and scalability challenges in large action spaces.

Overall, the comparative results in Figure 8 confirm that explicit agent coordination and context-aware reinforcement learning, as provided by MACxRL, are essential for
achieving sustainable, user-centric, and deployable smart-environment control, outperforming both traditional controllers and standard deep learning models.

## 6.9. Sensitivity Analysis

### 6.9.1. Impact of Reward-Weights

To assess the robustness and adaptability of the proposed MACxRL framework, we perform sensitivity analysis by varying energy weight $w_E \in \{0.45, 0.55, 0.65\}$, comfort preservation weight $w_c \in \{0.20, 0.30, 0.40\}$, and user satisfaction/stability weight $w_S \in \{0.15, 0.25, 0.35\}$. Our goal is to understand how these variations affect system behavior, which is essential for practical deployment. Table 8 reports the averaged results over 500 simulation steps and 10 independent runs when varying reward-weights.

<table><caption>**Table 8.** Impact of Different Reward-Weights on System Performance.</caption><thead><tr><th>Configuration</th><th>w<sub>E</sub></th><th>w<sub>c</sub></th><th>w<sub>S</sub></th><th>Normal. EC</th><th>TC</th><th>US/PRS</th></tr></thead><tbody><tr><td>Energy-priority</td><td>0.65</td><td>0.20</td><td>0.15</td><td>0.489 ± 0.01</td><td>0.82 ± 0.02</td><td>0.80 ± 0.02</td></tr><tr><td>Comfort-priority</td><td>0.45</td><td>0.40</td><td>0.15</td><td>0.512 ± 0.01</td><td>0.85 ± 0.01</td><td>0.84 ± 0.01</td></tr><tr><td>Stability-priority</td><td>0.50</td><td>0.25</td><td>0.25</td><td>0.548 ± 0.02</td><td>0.87 ± 0.01</td><td>0.83 ± 0.01</td></tr><tr><td>Balanced</td><td>0.55</td><td>0.30</td><td>0.15</td><td>0.523 ± 0.02</td><td>0.84 ± 0.01</td><td>0.86 ± 0.01</td></tr></tbody></table>

The sensitivity results reveal clear trade-offs among energy efficiency, comfort preservation, and policy stability. Increasing the energy weight $w_E$ leads to improved energy efficiency by encouraging more aggressive energy-saving actions. However, this comes at the cost of a slight degradation in thermal lighting comfort and overall user satisfaction. In contrast, increasing the comfort weight $w_C$ enhances thermal and lighting comfort levels by prioritizing indoor environmental quality. It also results in higher energy consumption, particularly during periods of low occupancy. Finally, increasing the stability weight $w_S$ improves policy smoothness and user satisfaction by reducing oscillatory control behavior and reward fluctuations, although it may marginally slow convergence due to more conservative action selection.

Among all tested setting, $w_E = 0.55$, $w_C = 0.30$ and $w_S = 0.15$ emerges as the best overall compromise, achieving near-optimal energy efficiency, high and stable comfort levels, low variance and fast convergence. This configuration delivers the highest aggregate performance across all three metrics, making it the most suitable choice for general-purpose smart-environment deployment. However, this configuration still requires manual tuning. An automatic weight-tuning mechanism is not yet integrated in the current system. As part of future work, we plan to incorporate an auto-calibration module capable of adjusting reward weights $w_1, w_2, w_3$ online based on performance feedback. This enhancement will allow the system to autonomously balance comfort and energy savings without manual intervention, thereby improving scalability and long-term adaptability.

### 6.9.2. Impact of Energy Threshold

To evaluate the robustness of MACxRL under different functional constraints, we analyze the impact of varying the energy budget threshold, which determines the maximum allowable energy usage during system operation. Three main thresholds are considered: 70% (strict constraint), 80% (balanced), and 90% (relaxed constraint). All results are averaged over 500 simulation steps and 10 independent runs. As shown in Table 9, varying the energy budget threshold introduces a clear trade-off between energy efficiency and user experience. Under a strict threshold (70%), MACxRL achieves the lowest normalized energy consumption, confirming its effectiveness in enforcing aggressive energy-saving policies, albeit with a slight reduction in comfort and user satisfaction. The 80% threshold
provides the most balanced performance, yielding high thermal-lighting comfort and US/PRS while maintaining strong energy efficiency, and is therefore adopted as the default setting. When the threshold is relaxed to 90%, comfort and satisfaction improve further at the expense of increased energy consumption. Overall, MACxRL adapts smoothly to different energy constraints while maintaining stable behavior.

<table><caption>**Table 9.** Impact of Energy Threshold on System Performance.</caption><thead><tr><th>Energy Threshold</th><th>Normal. EC</th><th>TC</th><th>US/PRS</th></tr></thead><tbody><tr><td>70% (Strict)</td><td>0.485 ± 0.01</td><td>0.81 ± 0.02</td><td>0.79 ± 0.02</td></tr><tr><td>80% (Balanced)</td><td>0.512 ± 0.01</td><td>0.85 ± 0.01</td><td>0.84 ± 0.01</td></tr><tr><td>90% (Relaxed)</td><td>0.548 ± 0.02</td><td>0.87 ± 0.01</td><td>0.86 ± 0.01</td></tr></tbody></table>

## 6.10. Convergence of MACxRL

Figure 9 illustrates the convergence behavior and robustness of the three evaluated control mechanisms across 80 simulation steps: the proposed MACxRL (Global MDP Q-learning), the no-coord approach [28], and the rule-based controller [29]. The results clearly indicate that MACxRL demonstrates a consistent upward trend with rewards stabilizing around 80, indicating effective learning of coordinated lighting, HVAC, and energy-management actions. In contrast, the no-coord approach [28] displays high volatility and inconsistent reward evolution. Rewards fluctuate widely between 40 and 70, demonstrating that independent decisions among lighting, HVAC, and energy subsystems lead to unstable behavior. The rule-based controller [29] is more stable than the no-coord approach [28] but still exhibits significant drops and fluctuations, particularly in dynamic temperature or occupancy transitions. Its fixed IF-THEN logic allows rapid decisions but lacks adaptability, causing rewards to oscillate between 40 and 70, depending on dynamic scenario conditions. Deep reinforcement learning approaches such as DQN [30] and Actor-Critic [31] demonstrate better learning capability, gradually increasing their cumulative reward, yet they still experience visible oscillations caused by delayed adaptation to contextual variations and lack of predictive coordination. LSTM-based approach [32], benefiting from temporal state awareness, shows smoother improvement, but requires higher computational resources and longer training to reach optimal reward performance.

<figure><img src="image_12.png" /><figcaption>**Figure 9.** Convergence performance comparison of MACxRL versus baseline approaches.</figcaption></figure>

Overall, MACxRL achieves the highest reward stability, the lowest variance, and the most robust convergence profile among the evaluated methods. This reflects the lightweight short-term context prediction, the strength of coordinated multi-agent decision-making and the ability of reinforcement learning to adapt policies over time.
## 6.11. Global Comparisons

Table 10 provides a global comparison of standard controllers, deep learning-based approaches, and the proposed MACxRL framework in terms of energy efficiency, comfort, stability, computational cost, and convergence speed. No-coord and rule-based controllers benefit from low computational cost and immediate execution but suffer from limited adaptability, low energy efficiency, and poor long-term stability, particularly in dynamic smart-environment scenarios requiring coordination and contextual awareness. Deep learning approaches such as LSTM and DQN/actor-critic improve energy efficiency and comfort by learning adaptive behaviors. However, they introduce higher computational overhead and slower convergence, while lacking explicit multi-agent coordination. MACxRL combines explicit context awareness, coordinated multi-agent decision-making, and lightweight reinforcement learning to achieve fast convergence, stable long-term behavior, and low computational cost, making it a practical, scalable, and deployable solution for sustainable and user-centric smart-environment control.

<table><caption>**Table 10.** Comparison of different approaches across different criterion.</caption><thead><tr><th>Criterion</th><th>No-Coord</th><th>Rule-Based</th><th>LSTM</th><th>DQN/AC</th><th>MACxRL</th></tr></thead><tbody><tr><td>Energy Efficiency</td><td>Low</td><td>Medium</td><td>High</td><td>High</td><td>High</td></tr><tr><td>Comfort</td><td>Low</td><td>Medium</td><td>High</td><td>High</td><td>High</td></tr><tr><td>Policy Stability</td><td>Low</td><td>Medium</td><td>High</td><td>High</td><td>High</td></tr><tr><td>Multi-Agent Coordination</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Context Awareness</td><td>✗</td><td>✗</td><td>Limited</td><td>Limited</td><td>Limited</td></tr><tr><td>Convergence Speed</td><td>-</td><td>Immediate</td><td>Moderate</td><td>Moderate</td><td>Fast</td></tr><tr><td>Computational Cost</td><td>Very Low</td><td>Very Low</td><td>High</td><td>High</td><td>Low</td></tr></tbody></table>

Although the proposed framework performs effectively under normal communication conditions, it still depends on periodic agent-to-agent data exchange, which may pose limitations when communication is unstable. If links fail, agents operate temporarily using the last known local state, but extended disconnection may reduce global optimality. To improve robustness, secure and communication-aware strategies are needed. Similarly, Liu et al. [34] highlight the importance of fault-tolerant cooperative control under FDI attacks. As future work, we aim to embed lightweight intrusion and anomaly detection modules for resilient operation under uncertain or adversarial conditions.

## 7. Conclusions

In this paper, we propose a framework based on context-aware predictive reasoning formalism for smart learning environments. This model is based on a Markov decision process, where users can autonomously manage their daily tasks and circular activities. We have deployed distributed agents to handle conflicting goals, which may cause context exchanges between different agents, advising them to gain maximal rewards by continuous reinforcement learning and then adapt their behavior accordingly.

We conduct extensive experiments using two widely used baselines. MACxRL framework achieves an energy consumption score (EC) of 0.526 and consistently outperforms traditional control models, demonstrating improvements in user satisfaction, comfort, and overall system responsiveness. As a model-agnostic, plug-and-play enhancement, MACxRL offers a practical and efficient solution for smart environments. However, the proposed framework relies on simplified context representations and predefined state-action abstractions, which may not fully capture the complexity of real-world environments. The MDP formulation assumes stable transition dynamics, making it less suited for highly rapidly changing contexts. Moreover, the use of rule-derived reward functions, while
effective for controlled simulations, may require domain-specific tuning to remain robust across diverse smart environments. In future work, we aim to extend this framework by integrating AI-assistive capabilities across a wider range of smart domains, incorporating richer context ontologies, and developing more adaptive AI agents capable of operating under greater environmental uncertainty.

**Author Contributions:** Conceptualization, A.A., B.A. and A.L.; methodology, A.A., B.A., and A.L.; software, A.L.; writing—original draft preparation, B.A., S.A. and A.L.; writing—review and editing, A.A., A.L., S.A. and P.R.; visualization, A.L.; supervision, A.A. and P.R. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Data Availability Statement:** The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.

**Acknowledgments:** This research was supported by the Center for Theoretical Colleges of Qassim University.

**Conflicts of Interest:** The authors declare no conflicts of interest.

## References

1. Perera, C.; Zaslavsky, A.; Christen, P.; Georgakopoulos, D. Context aware computing for the internet of things: A survey. *IEEE Commun. Surv. Tutor.* **2013**, *16*, 414–454. [CrossRef]

2. Baldauf, M.; Dustdar, S.; Rosenberg, F. A survey on context-aware systems. *Int. J. Ad Hoc Ubiquitous Comput.* **2007**, *2*, 263–277. [CrossRef]

3. Ye, J.; Coyle, L.; Dobson, S.; Nixon, P. Ontology-based models in pervasive computing systems. *Knowl. Eng. Rev.* **2007**, *22*, 315–347. [CrossRef]

4. Sutton, R.S.; Barto, A.G. *Reinforcement Learning: An Introduction*; MIT Press: Cambridge, MA, USA, 2018.

5. Wang, X.; Sandholm, T. Reinforcement learning to play an optimal Nash equilibrium in team Markov games. *Adv. Neural Inf. Process. Syst.* **2003**, *15*. Available online: https://proceedings.neurips.cc/paper_files/paper/2002/file/f8e59f4b2fe7c5705bf878bbd494ccdf-Paper.pdf (accessed on 5 January 2016).

6. Boukerche, A.; Ho, P. A reinforcement learning-based approach for smart home energy management. *J. Netw. Comput. Appl.* **2019**, *132*, 16–25.

7. Tesauro, G.; Jong, N.K.; Das, R.; Bennani, M.N. A hybrid reinforcement learning approach to autonomic resource allocation. In Proceedings of the IEEE International Conference on Autonomic Computing, Dublin, Ireland, 12–16 June 2006; pp. 65–73.

8. Du, H.; Thudumu, S.; Vasa, R.; Mouzakis, K. A survey on context-aware multi-agent systems: Techniques, challenges and future directions. *arXiv* **2024**, arXiv:2402.01968.

9. Chen, B.; Xu, M.; Liu, Z.; Li, L.; Zhao, D. Delay-aware multi-agent reinforcement learning for cooperative and competitive environments. *arXiv* **2020**, arXiv:2005.05441.

10. Chen, D.; Zhang, Q. Context-aware Bayesian network actor-critic methods for cooperative multi-agent reinforcement learning. In Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA, 23–29 July 2023; pp. 5327–5350.

11. Li, X.; Zhang, J. Context-aware communication for multi-agent reinforcement learning. *arXiv* **2023**, arXiv:2312.15600.

12. Iqbal, S.; Sha, F. Actor-attention-critic for multi-agent reinforcement learning. In Proceedings of the International Conference on machine learning, Long Beach, CA, USA, 9–15 June 2019; pp. 2961–2970.

13. Foerster, J.; Nardelli, N.; Farquhar, G.; Afouras, T.; Torr, P.H.; Kohli, P.; Whiteson, S. Stabilizing experience replay for deep multi-agent reinforcement learning. In Proceedings of the International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017; pp. 1146–1155.

14. Park, B.; Choi, J. Message Action Adapter Framework in Multi-Agent Reinforcement Learning. *Appl. Sci.* **2024**, *14*, 10079. [CrossRef]

15. Brenon, A.; Portet, F.; Vacher, M. Arcades: A deep model for adaptive decision making in voice-controlled smart-home. *Pervasive Mob. Comput.* **2018**, *49*, 92–110. [CrossRef]

16. Hosseinloo, A.H.; Dahleh, M.A. Event-Triggered Reinforcement Learning: An Application to Buildings’ Micro-Climate Control. In Proceedings of the AAAI Spring Symposium: MLPS, Stanford, CA, USA, 23–25 March 2020.

17. Kalivarathan, S.; Mohamed, M.A.R.; Ravikumar, A.; Harini, S. Intelligence of Things: A Spatial Context-Aware Control System for Smart Devices. *arXiv* **2025**, arXiv:2504.13942.
18. Chen, B.; Liu, Z.; Zhu, J.; Xu, M.; Ding, W.; Li, L.; Zhao, D. Context-aware safe reinforcement learning for non-stationary environments. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Xi'an, China, 30 May-5 June 2021; pp. 10689-10695.

19. Hallou, A.; Fissaa, T.; Hafiddi, H.; Nassar, M. Context-Aware IoT System Development Approach Based on Meta-Modeling and Reinforcement Learning: A Smart Home Case Study. *Int. J. Online Biomed. Eng.* **2024**, *20*, 25. [CrossRef]

20. Li, M.; Wu, Y. Intelligent control system of smart home for context awareness. *Int. J. Distrib. Sens. Netw.* **2022**, *18*, 15501329221082030. [CrossRef]

21. Weerasinghe, S.; Zaslavsky, A.; Loke, S.W.; Hassani, A.; Medvedev, A.; Abken, A. Adaptive context caching for IoT-based applications: A reinforcement learning approach. *Sensors* **2023**, *23*, 4767. [CrossRef]

22. Moradi, H.; Zamani, B.; Zamanifar, K. CaaSSET: A framework for model-driven development of context as a service. *Future Gener. Comput. Syst.* **2020**, *105*, 61-95. [CrossRef]

23. Delnevo, G.; Ghini, V.; Fiumana, E.; Mirri, S. A Support Tool for Emergency Management in Smart Campuses: Reference Architecture and Enhanced Web User Interfaces. *Sensors* **2024**, *24*, 5887. [CrossRef]

24. Polo-Rodríguez, A.; Fiorini, L.; Rovini, E.; Cavallo, F.; Medina-Quero, J. Enhancing smart environments with context-aware chatbots using large language models. *arXiv* **2025**, arXiv:2502.14469. [CrossRef]

25. Violos, J.; Mamanis, G.; Kompatsiaris, I.; Papadopoulos, S. Cognition and context-aware decision-making systems for a sustainable planet: A survey on recent advancements, applications and open challenges. *Discov. Sustain.* **2025**, *6*, 1-43. [CrossRef]

26. Augusto, J.C. Contexts and context-awareness revisited from an intelligent environments perspective. *Appl. Artif. Intell.* **2022**, *36*, 2008644. [CrossRef]

27. Karchoud, R.; Roosevelt, P.; Dalmau, M.; Illarramendi, A.; Ilarri, S. One app to rule them all: Collaborative injection of situations in an adaptable context-aware application. *J. Ambient Intell. Humaniz. Comput.* **2019**, *10*, 4679-4692. [CrossRef]

28. Afram, A.; Janabi-Sharifi, F. Theory and applications of HVAC control systems-A review of model predictive control (MPC). *Build. Environ.* **2014**, *72*, 343-355. [CrossRef]

29. Park, J.H.; Lee, S.; Kang, M. Development and evaluation of rule-based control strategies for smart building energy management. *Appl. Energy* **2020**, *264*, 114487.

30. Chifu, V.R.; Cioara, T.; Pop, C.B.; Rusu, H.G.; Anghel, I. Deep Q-learning-based smart scheduling of EVS for demand response in smart grids. *Appl. Sci.* **2024**, *14*, 1421.

31. Coraci, D.; Brandi, S.; Piscitelli, M.S.; Capozzoli, A. Online implementation of a soft actor-critic agent to enhance indoor temperature control and energy efficiency in buildings. *Energies* **2021**, *14*, 997. [CrossRef]

32. Anan, M.; Kanaan, K.; Benhaddou, D.; Nasser, N.; Qolomany, B.; Talei, H.; Sawalmeh, A. Occupant-aware energy consumption prediction in smart buildings using a LSTM model and time series data. *Energies* **2024**, *17*, 6451. [CrossRef]

33. Annane, B.; Alti, A.; Osman, G. SecNetworkCloudSim: An extensible simulation tool for secure distributed mobile applications. *Int. J. Commun. Netw. Inf. Secur.* **2020**, *12*, 47-62.

34. Liu, G.; Sun, Q.; Su, H.; Wang, M. Adaptive cooperative fault-tolerant control for output-constrained nonlinear multi-agent systems under stochastic FDI attacks. *IEEE Trans. Circuits Syst. I Regul. Pap.* **2025**, *72*, 6025-6036. [CrossRef]

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.