# From Waves to Words: Large Perceptive Models as the Future of IoT and RF Intelligence

Lina Bariah, Thomas Mongaillard, Samson Lasaulce, Wassim Hamidouche, Hang Zou, Mérouane Debbah

► To cite this version:

Lina Bariah, Thomas Mongaillard, Samson Lasaulce, Wassim Hamidouche, Hang Zou, et al.. From Waves to Words: Large Perceptive Models as the Future of IoT and RF Intelligence. IEEE Wireless Communications, In press, 10.1109/MWC.2025.3613680. [hal-05388155](https://hal.science/hal-05388155v1)

HAL Id: hal-05388155

https://hal.science/hal-05388155v1

Submitted on 28 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# From Waves to Words: Large Perceptive Models as the Future of IoT and RF Intelligence

Lina Bariah, Thomas Mongaillard, Samson Lasaulce, Wassim Hamidouche, Hang Zou, and Merouane Debbah

***Abstract***—As the physical and digital worlds are converging with the rapid evolution of advanced technologies, the Internet-of-Things (IoT) is transitioning from a network of passive, rule-based nodes into an intelligent, adaptive system, that understands, reasons, and responds to human intent. The future of IoT is foreseen to be systems that not only listens but also understands, where human demands are seamlessly translated into machine actions, and sensor data are transformed into meaningful insights. At the core of this vision is the significant advancements of Large Language Models, which offer a bridge between human language and machine language, and support the fusion of multimodal data streams. In this article, we put a forward looking vision of how large language models (LLMs) will be embedded into the fabric of IoT systems, represented by the Large Ambient Models (LPMs) concept, for which users will no longer configure devices manually but simply express their needs in their natural language, leaving the role of translation and orchestration to the LLMs, to optimize in real-time, enabling use-cases from network resource allocation and environment mapping, into energy optimization. Furthermore, we explore how LPMs can understand the non-trivial language of IoT signals, transforming radio-frequency (RF) data, environmental metrics, and sensor outputs into meaningful, proactive insights, that can be comprehended by the users. To realize this vision, we introduce an agentic LLM-IoT architecture, where specialized estimation, modeling, and solving agents collaborate to automate decision-making, optimize performance, and enhance system intelligence.

## INTRODUCTION

The Internet of Things (IoT) has evolved significantly since its emergence as a technology aimed at connecting industrial machines to the internet for enhanced operational efficiency. While the term was coined in 1999 following the invention of radio-frequency identification (RFID) for supply chain tracking, the concept of IoT dates back to early 1980s, when advancements in microprocessor technology enabled the connection of devices to the internet, enabling a swarm of industrial applications for automation, monitoring, and control purposes [1]. After that, the IoT technology has transitioned into more user-centeric applications with the introduction of the smart home devices idea, introducting connected thermostats, smart lighting systems, wearable devices, and personalized healthcare tools, among others.

IoT has evolved from its industrial-focused origin in the 1980s, where connected devices enabled automation and monitoring, to a broader ecosystem driven by RFID-based supply

chain tracking. After that, it has transitioned into more user-centric applications, introducting connected thermostats, smart lighting systems, wearable devices, and personalized healthcare tools, among others. At the heart of the IoT evolution lies the advancement of wireless communication technologies, which have been the key element in the development of the IoT paradigm. From the 3rd generation (3G), which provided the foundation for mobile internet access, enabling basic IoT applications with limited data transmission capabilities, to 4G with higher data rates and lower latency, facilitating more complex IoT applications, including real-time video streaming and data analytics. Lastly, 5G networks, paralled with advanced cloud and edge computing technologies, introduced ultra-low latency and improved capacity to connect a massive number of devices simultaneously. This all have been reinforced with the integration of artificial intelligence (AI), allowing advanced tasks such as optimization, pattern recognition, smart sensing, and localization. Within the context of AI, Machine Learning (ML) and deep learning (DL) have been widely applied to IoT scenarios [2], [3], enabling IoT systems to process large volumes of data and make intelligent decisions in real-time. While having some merit, most ML/DL models are trained on task-specific datasets, which limits their ability to generalize to new and unseen scenarios. Subsequently, maintaining and updating these models across IoT devices would be impractical due to the significant computational and financial costs involved. This challenge is further pronounced by the fact that IoT devices and networks are highly heterogeneous, with varying hardware capabilities, data formats, and communication protocols. For example, the same type of sensor might generate data at different resolutions or sampling rates depending on the hardware and vendor. This heterogeneity introduces challenges in ensuring the compatibility and scalability of conventional DL techniques, which often assume homogeneous and well-structured input data. As a result, models trained on one set of IoT devices may fail when deployed on another set with different characteristics. Finally, the dynamic and distributed nature of IoT environments adds another layer of complexity.

Despite the unique blend of AI and wireless technologies that have led to significant advancements in IoT, current IoT systems have yet to fully achieve a human-centric approach. In particular, current AI-driven IoT systems have the following limitations:

* **Uniform user experience:** IoT devices are often constrained by predefined optimization chains and manufacturer-imposed design limitations. This results in

<footnote>L. Bariah, S. Lasaulce, W. Hamidouche, and M. Debbah are with the KU 6G Research Center, Department of Computer and Information Engineering, Khalifa University, Abu Dhabi 127788, UAE. T. Mongaillard is with Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France. H. Zou is with the Technology Innovation Institute, Abu Dhabi, UAE.</footnote>
a lack of flexibility to adapt to individual user demands, offering a one-size-fits-all approach that limits personalization.

*   **Lack of natural language interactions:** Most IoT systems rely on predefined or static command interfaces or mobile applications for interaction, which typically require users to search through menus, configure settings manually, or use specific, predefined commands to perform tasks. Accordingly, such systems are not designed to support machine-user communication through natural language, making them less user-friendly and harder to interact with.

*   **Limited generalizability:** IoT solutions are often designed for specific tasks or contexts. Even AI-driven IoT solutions are constrained by the specific scenarios or datasets they are trained on. This limits their ability to generalize to new tasks, environments, or unforeseen user demands.

*   **Underutilization of IoT signal potential:** IoT devices generate large amounts of meaningful, multimodal data, ranging from sensor readings to spatio-temporal information, that often go underutilized. Current systems fail to extract meaningful insights from this data or leverage it for context-aware applications.

The emergence of large-scale generative AI (GenAI) models, including large language models (LLMs), have motivated the research community to revisit these limitations and explore new opportunities that can be unlocked through their advanced capabilities. LLMs are a subset of large-scale GenAI systems, designed to process, understand, and generate human-like text by leveraging deep neural networks trained on massive datasets. These models, such as GPT, BERT, and PaLM, are built using billions, or even trillions, of parameters, enabling them to capture language patterns, context, and semantics. The ability of these models to generalize across domains, handle diverse tasks, and adapt to previously unseen scenarios makes them critical to advancing human-centric applications, including conversational systems and multimodal integrations.

Emphasizing on the aforementioned challenges brought by current IoT systems, LLMs can address these limitations by interpreting user intent and translating it into machine commands. Rather than requiring manufacturers to provide exhaustive high-level controls, they can focus on generic, low-level functionalities, such as adjusting voltage, and let LLMs handle the rest. This flexibility allows LLMs to design controls that match users' expectations, without requiring users to define every detail themselves. By bridging the communication gap, LLMs make it possible for users to interact with their surroundings in a more natural, adaptive way. On the other side, i.e., IoT signals-to-humans side, LLMs can transform how IoT-generated data is interpreted and communicated to users. Instead of overwhelming the users with raw, complex sensor data, LLMs can process IoT signals, extract useful insights, and communicate them to the users in natural language.

### A. Why is this shift important?

Empowering users enhances the quality of their experience by providing greater control. Flexibility is crucial because every environment and user exhibit unique characteristics. By enabling users to tailor their smart environments to their specific needs, LLMs can optimize costs, energy consumption, and data efficiency, making these environments not only more responsive but also more sustainable and personalized. This flexibility becomes even more valuable when IoT systems move beyond merely reacting to user requests and actively communicate with them. As noted, IoT devices generate vast amounts of multimodal, rich data. However, much of this data remains underutilized because it is often presented in raw, technical formats that are difficult for users to intuitively interpret. This is where the true potential of LLMs becomes evident.

Despite significant progress, several challenges persist. How can we accurately interpret user intent? What strategies can ensure seamless communication between devices, particularly as the number of smart devices continues to expand? Integrating LLMs into multi-user households, developing sustainable control mechanisms, and efficiently managing power supplies are complex issues that demand innovative solutions. Addressing these challenges is crucial to unlocking the full potential of LLMs within IoT systems.

### B. State-of-the-Art

Although still a relatively new area of research, several studies have already explored the integration of LLMs with IoT systems. The authors in [4] proposed an open-source, multisensory large language model designed for IoT systems, which includes curating a comprehensive IoT dataset and introducing a new multisensory multitask adapter layer. The authors in [5] investigated the role of LLMs in IoT, with a particular focus on their reasoning capabilities. It also explores LLM integration with edge, fog, and cloud computing to optimize resource utilization and support real-time processing. Lastly, the authors in [6] introduced a framework that enhances LLMs' perception and reasoning for IoT tasks by integrating sensor data, specialized prompting, and retrieval-augmented generation (RAG).

From a different perspective, several contributions have been made to study the role of LLMs in understanding, formulating, and solving optimization problems, with an increasing focus on interpreting human intent in user-IoT interactions. Estimation methods, such as those in [7], extract optimization parameters from natural language, though they suffer from error accumulation and limited scalability. Modeling approaches, like in [8], translate mathematical word problems into structured optimization formulations. Additionally, [9] integrates both modeling and estimation with built-in error correction before invoking solvers.

### C. Contributions

Unlike existing literature, this article introduces the concept of large perceptive models (LPMs), representing a holistic
<figure><img src="image_2.png" /><figcaption>Fig. 1. A Two-Layer Large Perceptive Model (LPM) System for IoT Interpretation and Optimization</figcaption></figure>

vision of how LLMs can drive radical advancements in the future of IoT systems. We aim to bridge the gap between human intent and machine execution, as well as transform complex IoT signals into meaningful, user-friendly insights. In essence, we investigate how LLMs can evolve IoT from a reactive, manually configured system into an intelligent, human-centered, and adaptive network, where LLMs seamlessly understand, optimize, and interact with both IoT devices and users. The contributions of this article are:

* We propose a new paradigm, an *LLM-empowered IoT environment*, where a large *perceptive* model serves as both the voice of IoT data and the brain translating user requests into actions.
* We further introduce an agent-based LLM-IoT architecture, where specialized LLM agents (Modeling, Estimation, and Solving Agents) collaborate to automate decision-making and optimize system performance.
* We propose a use case of smart home energy management, that demonstrate how user-IoT interactions can be realized through the integration of LLMs.

# I. LLMs IN IOT: BRIDGING THE GAP BETWEEN SIGNALS AND LANGUAGE

IoT systems are characterized by the vast amount of data they generate, which represents a continuous flow of information from various sources, including sensors, devices, embedded systems, and connected platforms. This data encompasses a wide range of types, such as environmental metrics (e.g., temperature, humidity, and air quality), spatial and motion data, network performance statistics, and user activity patterns. While this data is rich in nature, effectively understanding

and utilizing it typically requires technical experts proficient in the specialized 'language' of IoT. These experts often focus on specific subsets of information relevant to predefined tasks, limiting the broader potential of IoT systems. As a result, the value offered by current IoT systems falls short of their envisioned capabilities, highlighting the need for the integration of LLMs into the IoT ecosystem.

LLMs enhance IoT communication by enabling comprehensive analyses, providing context-aware and proactive recommendations, and leveraging the multimodal nature of the data, rather than simply presenting raw data or generic notifications. For example, in a wireless system, instead of merely sending a notification like "Low signal strength detected" (or its equivalent in system language), an LLM can generate a more informative message: "The Wi-Fi signal in the office is weak due to interference from a newly installed device. Switching to Channel 6 is recommended to improve connectivity." This approach empowers users to take informed action without the need to manually interpret complex dashboards or decipher cryptic system alerts.

## A. Large Perceptive Models

The proposed paradigm of Large Perceptive Models (LPMs) represents a significant shift toward a multi-agent framework that enhances how we interact with the physical world, and vice versa. The notion of the LPM concept is twofold (Fig. 1). First, LLMs act as a bridge to make the physical world "speak", translating diverse data sources into a human-readable insights ("LLMs as interpreters"). This framework comprises two parts, one or more agents dedicated to aligning and understanding multimodal data, and a second agent,
often a language model, responsible for facilitating human communication (which can be through textual messages, audio signals, etc.). The other side of the LPM concept, *LLMs as optimizers*, depends on employing LLMs to facilitate human-to-IoT communication, where users can express high-level requests in natural language. Through a multi-agent architecture, LLM agents can extract human intents from voice/textual requests, and then an LLM agent formulates the corresponding optimization or control problem, selects the appropriate mathematical framework, and computes the optimal solution tailored to the user's personal preferences and constraints. The key idea of relying on the agentic architecture is that optimization problems are generally complex, multi-step tasks, that requires dedicated agents per each, hence, using a multi-agent framework can help to convert complex tasks into multiple manageable steps.

The following elements outline how IoT systems can effectively communicate with users in the context of LLM integration:

**Generalizability:** Conventional IoT systems are designed for narrow-scoped tasks with fixed parameters, limiting their ability to adapt to new devices, environments, or user behaviors without manual reconfiguration or re-tuning. For example, a system in a smart factory can be designed to switch channels when signal interference exceeds a certain threshold. However, when moved to a dynamic outdoor deployment with mobile nodes and random interference sources, such systems may fail. LLMs exhibit remarkable generalization capabilities, enabling them to perform effectively across a wide range of tasks beyond their initial training data.

**Natural language interaction:** LLMs enable IoT systems to understand and respond to human language naturally, without the need for structure commands. For instance, if you're experiencing an issue with your connectivity, system logs will show something around "RSSI: -75 dBm, SNR: 10 dB, Packet loss: 12%", for which a regular user will not understand, or find a way to fix the network issue. An LLM can help analyzing the RF environment, extracting the root cause, and communicating the solution in a more user-friendly way, e.g., "The RF signal near your smart door lock is unstable due to metal interference. Moving the router closer or repositioning the lock may improve connectivity."

**Advanced Reasoning Capabilities:** LLMs can reason across multiple variables (and data modalities) and historical context to generate more insightful outputs. Techniques such as Chain-of-Thought (CoT) prompting encourage models to generate step-by-step explanations, enhancing their performance on complex tasks like arithmetic and logical reasoning. Also, it was shown that LLMs excel in inductive reasoning (drawing generalized conclusions from specific examples) [10]. This ability to reason across time, context, and sensor data enables more intelligent and human-like decision-making in IoT environments.

**Human-centric applications:** The aim of incorporating LLMs into IoT systems is to shift from pre-programmed interactions to adaptive, real-time user-driven engagement. Within this

context, IoT devices can adapt dynamically to user's demands, while ensuring that users can simply express their needs in natural language, instead of manually configuring settings. In addition to this, LLMs help the users to interact with multiple IoT devices simultaneously without the need for separate apps, imposing his/her own constraints and the elements to be optimized according to his/her needs. Considering a wireless resource management scenario, users express their requirements in natural language, such as "I'm hosting an online conference, and I need my video calls to be smooth and uninterrupted. At the same time, I don't want to block other colleagues' access to streaming services and IoT devices. Can you optimize the network for this?". Using the user request, for which an LLM will translate the user's request into a mathematical optimization problem that maximizes network performance for high-priority applications while maintaining overall fairness in resource distribution.

**Multi-modal IoTs:** Unlike traditional IoT systems, which generally rely on a single-modal data streams (for multi-modality readings, system will process each modality separately), future LLM-driven systems are anticipated to realize the fusion of multiple data streams with a different modality nature, a combination of RF signals, sensor data, environmental metrics, motion detection, and even audio-visual inputs. It should be reminded that, events are rarey happen due to a single reason, but rather, due to a blend of multiple factors from the environment and the objects in it, and hence, extracting the most useful insight would be driven by information retrieved from all these factors, in the IoT case, these factors will be images from cameras, sounds from speakers, smells from sensors, etc. To further emphasize the importance of multi-modal incorporation, take this LLM example: in traditional IoT systems, a smart security camera detects motion and sends an alert, "Movement detected. Time Stamp: 3 AM". However, when integrating motion data with RF signal strength and environmental noise, an LLM output can be: "Movement detected at 3 AM, but RF signals indicate it's your pet. No security action needed". By grounding AI models in these diverse data modalities, we bridge the gap between the physical and digital worlds, leading to more intelligent and adaptive IoT environments [11].

## B. Use-Cases: LLM-enhanced perception and interaction in IoT

1) **RF Signal understanding:** In conventional systems, RF signal understanding, for the sake of extract meaningful information about the environment or the devices within it, has been achieved through signal processing techniques, which analyze RF signals parameters, such as frequency, amplitude, and phase. With the integration of LLMs into IoT systems, there is a promising potential to enhance RF signals interpretation, leading to significant advancements in wireless network functionalities as well as environment understanding, including beamforming, resource allocations, etc. Traditionally, these tasks rely on predefined algorithms and fixed configurations, which may not adapt to dynamic environments. Within this
context, LLMs can be trained to interpret RF signal data by aligning it with descriptive natural language inputs. For example, particular sensors can be utilized to capture textual descriptions of the environment, such as object locations. LLMs can then fuse this language data with RF signals, providing more contextual understanding and compensating for missing details due to incomplete RF data, thereby enhancing overall sensing capabilities [12].

Furthermore, when considering the integration of RF data with multimodal inputs through LLMs, IoT systems gain a more comprehensive understanding of their environments. For example, combining RF signal data with visual inputs can enhance object detection capabilities, and can facilitate the IoT device to convey any relevant messages, extracted from the object detection process, to user in natural language [4].

2) Environment mapping: Environment mapping involves the reconstruction of high-resolution spatial representations that contain the positions and relationships between the objects within a particular area. In the context of IoT systems, such mapping is essential for facilitating autonomous navigation, optimizing resource distribution, and enabling context-aware interactions. For instance, in robotics, accurate environment maps enable robots to navigate complex environments autonomously.

The integration of LLMs into environment mapping tasks constitutes a significant advancement in this field. In particular, LLMs can combine data from a number of sensors, including RF signals, LiDAR, cameras, etc., to construct comprehensive and dynamic maps. By interpreting this multimodal data, LLMs can generate spatial representations that capture the physical layout, as well as the dynamic activities within the environment. It should be noted that, LLM-driven environment mapping can introduce tangible advancements to wireless networks, in terms of resource allocation, interference management, optimized routing protocols, to name a few.

3) Intent-driven optimization: Intent-driven optimization focuses on adjusting IoT system behaviors according to the user's intentions, which are expressed by natural language. LLMs play a key role in interpreting these intentions and translating them into actionable configurations for IoT devices [13]. In resource management tasks, LLMs can optimize the energy consumption or network bandwidth by analyzing user-defined priorities, ensuring optimized user experience. Within this context, personalized experiences are enhanced as IoT systems tailor their responses to individual user's preferences.

## II. AGENTIC ARCHITECTURE FOR IoT: THE NEXT FRONTIER IN IoT OPTIMIZATION AND CONTROL

One key observation upon which the approach detailed in this section is based is that many IoT technical problems e.g., control, communication, resource allocation, or estimation problems, can be solved according to a 3-step methodology: 1. determine a mathematical model of the problem under consideration; 2. estimate the model parameters; 3. exploit the model to achieve a given goal. Based on this approach, which is largely exploited by engineers, we propose an agentic

architecture with three agents: an LLM-based modeling agent (Agent 1); an LLM-based estimation agent (Agent 2); an LLM-based decision-making agent (Agent 3). For instance, this architecture is very well suited in the smart grid and for home energy management problems. In Sec. III, we develop the case of home energy management. For this case, the final goal is to produce a sequence of consumption power levels (Agent 3). For this purpose, the user is assumed to give a voice/text request to the energy management system, which is converted by Agent 1 into a suited model (e.g., an optimization problem or a dynamical equation). Agent 2 then extracts knowledge from the request and its own knowledge to estimate the model parameters. Below, we provide more details about the role and possible implementation of such agents in IoT networks.

### A. Modeling agent

The modeling agent serves as a bridge between natural language user requests and their mathematical representations. Its primary task is to convert user intentions into formal mathematical descriptions that can be processed by other agents in the system. While existing approaches focus on translating complete mathematical problem descriptions into formal optimization problems [9], our modeling agent faces a more challenging task: it must work with incomplete information derived from user intent. Rather than processing detailed mathematical descriptions, it operates on natural language requests that often lack explicit parameters, constraints, and technical specifications. This represents a significant advancement in the field, as it bridges the gap between casual user expressions and formal mathematical formulations [14].

When processing a user request, the modeling agent's output is typically formulated as an equation where the desired information or control signal appears as the result. For instance, in a control request scenario, the agent might need to determine power consumption vectors for one or multiple devices, while for information requests, it could be tasked with calculating specific values derived from sensor data or system states.

The agent's strength lies in its ability to understand implicit requirements and constraints without requiring technical expertise from users. This abstraction allows users to express their needs in natural language while the agent handles the complexity of mathematical modeling.

A key consideration in the modeling process is parameter accessibility. The agent is designed to only include parameters that are either available through the system's sensors and devices or explicitly provided by the user. For example, when modeling energy expenditure during a weightlifting exercise using smartwatch accelerometer data, the agent would include dumbbell weight as a parameter only if this information is provided in the user's request, as it cannot be measured by the available sensors.

### B. Estimation agent

The estimation agent (Agent 2) serves as the interface between IoT devices and the overall system, responsible for gathering
<figure><img src="image_3.png" /><figcaption>Fig. 2. Agentic architecture for IoT Systems: An Application to EV Charging Scenario</figcaption></figure>

ering and processing device data to support user queries and system operations. Therefore, estimation operations may have to be performed both from user queries and measurements. Recent research has demonstrated that LLMs can effectively analyze raw sensor data for certain tasks [15], [16], expanding their potential role in IoT systems. To maintain flexibility while ensuring optimal performance across different devices, manufacturers can provide structured prompts that specify accessible device parameters and standardized response formats. The estimator's role varies depending on the task at hand. For some operations, it leverages LLMs' capabilities to directly analyze sensor data and measurements. However, for specialized tasks where established algorithms already exist - such as human activity recognition from security camera feeds - Agent 2 interfaces with these proven solutions rather than attempting direct analysis. In such cases, Agent 2 works with the processed outputs from these specialized algorithms, integrating them into the broader system response. For example, when a user queries about home occupancy, the modeling agent might determine that security camera data is relevant. Instead of analyzing the video stream directly, the estimator would access the results from existing occupancy detection algorithms, allowing it to provide accurate information while leveraging established solutions. This hybrid approach combines the flexibility of LLM-based analysis with the efficiency of specialized algorithms, ensuring optimal performance across different types of IoT tasks. Concerning user queries, Agent 2 may have to extract some parameters which are necessary for control devices to be able to make decisions. For instance, this might be the number of devices to be switched off, the time at which to start or stop executing the task, etc. The literature on LLMs clearly demonstrates the capability of LLMs to perform such information extraction

operations [8]. In general, Agent 2 has to provide an estimate for any parameter which is required by Agent 3 to make its decision.

### C. Decision-making agent

Users want to control their devices in a way that it maximizes some performance metric. This performance metric is specified in the request, explicitly or implicitly and must be understood by Agent 1. It can be the electricity cost, the energy consumed, the duration of the control, the lifetime of the device, etc. This kind of control is most of the time the solution of an optimization problem or an optimal control problem. To maintain the flexibility of the design, we need to be able to solve any OP or OCP that will be given by the modeler. The goal of the decision-making agent is to run the solving process given the mathematical formulation from the modeler.

The decision-making agent serves as the mathematical problem-solving core of the system, specifically designed to handle fully formulated optimization problems and optimal control problems that come from the modeler agent. These problems are presented in a complete mathematical form, including objective functions, constraints, and all necessary parameters. For optimization problems, this typically includes the cost function to be minimized or maximized, equality and inequality constraints, and boundary conditions. In the case of optimal control problems, the agent receives the system dynamics equations, control constraints, initial and final conditions, and performance criteria. The solving agent's primary responsibility is to find solutions that satisfy all constraints while optimizing the given objective function. This task requires sophisticated numerical methods and can involve various mathematical techniques depending on the problem class - from linear and convex optimization to more
complex nonlinear and dynamic optimization scenarios. We
have identified two approaches for implementing this agent
by using LLM capabilities.

a) *Classic numerical solvers:* The agent can be im-
plemented to leverage existing optimization libraries through
a code interpreter tool. This approach allows direct use of
established solvers like HiGHS or Gurobi through Python
interfaces such as AMPL. While these solvers provide robust
solutions, they may sometimes yield suboptimal results due
to the complexity of IoT optimization problems. How can
the LLM capabilities can be exploited in such a context?
Remarkably, even for this standard approach, capabilities of
the LLM Agent 3 can be very useful: it can choose the best
numerical solver to be used for the OP selected by Agents 1
and 2; since most numerical solvers are iterative, Agent 3 can
tune the iterative algorithm parameters e.g., its initial solution
or its learning rate.

b) *Pure LLM approach:* An alternative implementation can be to give the OP to the LLM (Agent 3) and ask for a solution. At the time this paper is written, even LLMs like ChatGPT, Gemini, Llama, Mistral, Falcon, or more recently DeepSeek are not designed to deal with numbers. In particular, the LLM tokenizer has a limited vocabulary, which prevents vector of real numbers from being to be processed accurately. However, recent results have shown that LLMs can use knowledge from pre-training, its ability to analyze patterns in sequence of numbers, and its "reasoning" ability to produce coherent solutions. Although this research is nascent and would require addressing issues such as scalability (e.g., in terms of the dimensionality of the vector of variables in the function to be minimized), it currently provides promising results and demonstrates that at least rough solutions can be obtained for small-dimensional settings.

The decision-making agent primarily relies on the Code
Interpreter tool, which is essential for its operation. This tool
provides a runtime environment for executing classical solvers,
enabling direct interaction with optimization libraries. The
Code Interpreter allows the agent to run optimization algo-
rithms, monitor their performance in real-time, and perform
necessary debugging when numerical issues arise.

III. SMART HOME ENERGY MANAGEMENT CASE STUDY

For simplicity, following [14], we assume here that there
is only one electric device to be controlled (e.g., the EV
charging). The EV user is assumed to make a voice/text
request to his/her vehicle e.g., through his/her smartphone
or to the vehicle directly. From the sole knowledge of this
request, the LLM-based agentic architecture has to produce or
plan a sequence of charging power levels x = (x₁, x₂, ..., x_T)
to be applied to the EV battery, with ∀t ∈ {1, ..., T}, 0 ≤
x_min ≤ x_t ≤ x_max, T being the number of time-slots over
which the power is scheduled. First, let us describe a possible
implementation of the three agents introduced in Sec. II.
Second, let us discuss its performance when applied a given
benchmark.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Planned power charging vector produced by the 3-agents architecture for the EV charging scenario</figcaption></figure>

A simple way of implementing Agent 1 is to define it as
a classifier which has to select among a list of OP types the
most suited OP type (linear program, quadratic program, mini-
max class, convex program, linear minimum-time, discrete-
time linear quadratic regulator). The role of Agent 2 is then
to estimate the parameters of the selected OP. To this end, it
has to extract information from the user request (e.g., the time
at which it has to stop charging), from its own knowledge
base (e.g., the maximum value for the maximum charging
power Pmax) and from external databases (e.g., from the
Internet). A possible information extraction scheme which is
based on hard prompt engineering is proposed in [14]. For
Agent 3, a simple scheme is to assign to each OP a given
iterative numerical solver and prompt the Agent 3 to choose
the best initial solution and learning rate. For the EVRQ
dataset proposed by [14], Fig. 3 represents the power charging
vector produced by Agent 3 when tested for four different
requests (constant power, peak power minimization, charging
monetary cost, minimum charging time) from EVRQ. The
results are seen to fully consistent with the electricity price
and non-EV consumption. More generally, such an agentic
structure can be seen to reach relatively high accuracy in terms
of recognozing the user's intent, although the proposed scheme
may be improved by using fine-tuning and more OP classes.

IV. CHALLENGES AND FUTURE DIRECTIONS

A. Edge Generation

Running LLMs in real-time on edge devices presents a
significant challenge in latency and computational cost, due
to the resource constraints of IoT devices. Different than
cloud-based inferencing, which rather enjoys the availability of
high-performance GPUs/TPUs to handle large-scale models,
edge devices often rely on low-power processors with limited
memory, rendering LLM inference a bottleneck. This is further
pronounced with the recent studies demonstrating that more
resources need to be devoted to inferencing, to enable ad-
vance tasks, including reasoning, mathematical modeling, and
coding, which are essential in the context of IoT. Accordingly,
optimizing LLM deployment for edge devices will necessitate
a balance between model efficiency, optimized hardware,
and adaptive offloading strategies, in order to ensure real-time processing without compromising performance or energy efficiency.

## B. Communication

Communication in LLM-IoT systems faces the fundamental challenge of enabling effective interaction between LLMs and devices. Recently, AI companies have developed protocols allowing LLMs to interact with the external world through tools, requiring function calls in response to particular requests. However, a more universal solution has emerged with the Model Context Protocol (MCP), developed by Anthropic AI. MCP provides a universal way of making LLMs interact with other services, allowing them to process predesigned actions on databases, files, web, memory, and social networks. For IoT device communication, the well-known MQTT protocol (Message Queuing Telemetry Transport) can be utilized. This publisher-subscriber transport protocol enables devices to publish measures on a local MQTT broker to which agents can subscribe to retrieve sensor measurements.

## C. Multi-user adaptation

The primary challenge in multi-user adaptation in LLM-enhanced IoT lies in managing concurrent user interactions within shared spaces, where multiple individuals with different preferences and priorities simultaneously interact with the same IoT devices, requiring a sophisticated priority system to manage these conflicting demands. Another challenge involves the LLM's ability to accurately interpret and route user requests to the correct resources. When users make requests, the LLM must first establish the correct mapping between the user and their associated spaces, devices, and parameters.

## D. Tokenizing and Representing Physical Data for LLMs

Apart from the fact that LPMs are anticipated to align multimodal data streams (semantically and temporally), successfully representing IoT data for LLMs to process is a big challenge. This is stemmed from the fact that IoT data are generally raw numerical values, that lack the textual context, limiting the potential of LLMs in such applications. Accordingly, using conventional tokenization methods (that are considered for textual data), results in irregular tokenization of floating-point numbers, which adds complexity to arithmetic operations [6]. As a sequence, to utilize the full potential of LLMs within the context of IoT, new tokenization methods or sensory data prepossessing approaches are needed to ensure that LLMs can effectively interpret and analyze such kind of data, ensuring accurate representation of the physical data.

## V. CONCLUSION

The future of IoT systems is anticipated to witness a ubiquitous deployment of LLMs to assist in our daily life, not only allowing us to seamlessly talk to our environment, but also receive more useful context from our surroundings. In this article, we have laid the foundation for incorporating LLMs into the realm of IoT, where we introduced the concept

of LPMs, where LPMs are aimed to act as the translating interface between humans and the surrounding environment (with all kind of sensed data). From the other perspective, LPMs will enable us to communicate with the IoT devices in our environment, imposing our own constraints and maximizing our experiences.

## REFERENCES

[1] K. Rose, S. Eldridge, and L. Chapin, "The internet of things: An overview," *The internet society (ISOC)*, vol. 80, no. 15, pp. 1–53, 2015.

[2] J. Bian et al., "Machine learning in real-time internet of things (iot) systems: A survey," *IEEE Internet of Things Journal*, vol. 9, no. 11, pp. 8364–8386, 2022.

[3] H. Park et al., "Machine learning for internet of things: Applications and discussions," in *2024 International Conference on Artificial Intelligence in Information and Communication (ICAIIC)*, 2024, pp. 459–462.

[4] S. Mo, R. Salakhutdinov, L.-P. Morency, and P. P. Liang, "IoT-LM: Large Multisensory Language Models for the Internet of Things," Jul. 2024.

[5] I. Kök, O. Demirci, and S. Özdemir, "When IoT Meet LLMs: Applications and Challenges," in *2024 IEEE International Conference on Big Data (BigData)*. IEEE, 2024, pp. 7075–7084.

[6] T. An, Y. Zhou, H. Zou, and J. Yang, "IoT-LLM: Enhancing Real-World IoT Task Reasoning with Large Language Models," Oct. 2024.

[7] R. Ramamonjison, T. T. Yu, R. Li, H. Li, G. Carenini, B. Ghaddar, S. He, M. Mostajabdaveh, A. Banitalebi-Dehkordi, Z. Zhou, and Y. Zhang, "NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions," Mar. 2023.

[8] Z. Tang et al., "Orlm: Training large language models for optimization modeling," *arXiv preprint arXiv:2405.17743*, 2024.

[9] A. AhmadiTeshnizi, W. Gao, H. Brunborg, S. Talaei, and M. Udell, "OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale," Jul. 2024.

[10] X. Tang et al., "Large language models are in-context semantic reasoners rather than symbolic reasoners," *arXiv preprint arXiv:2305.14825*, 2023.

[11] L. Bariah and M. Debbah, "AI Embodiment Through 6G: Shaping the Future of AGI," *IEEE Wireless Commun.*, vol. 31, no. 5, pp. 174–181, 2024.

[12] L. Wang, C. Zhang, Q. Zhao, H. Zou, S. Lasaulce, G. Valenzise, Z. He, and M. Debbah, "Generative AI for RF Sensing in IoT systems," https://arxiv.org/abs/2407.07506v1, Jul. 2024.

[13] A. Sheriff, "Harnessing the power of intent-driven internet of llm agents (IIOAs) for enterprise it," Jul. 2024. [Online]. Available: https://outshift.cisco.com/blog/intent-driven-llm-agents-enterprise-IT?utm_source=

[14] T. Mongaillard, S. Lasaulce, O. Hicheur, C. Zhang, L. Bariah, V. S. Varma, H. Zou, Q. Zhao, and M. Debbah, "Large Language Models for Power Scheduling: A User-Centric Approach," Jun. 2024.

[15] Z. Li, S. Deldari, L. Chen, H. Xue, and F. D. Salim, "SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition," Oct. 2024.

[16] A. Russell-Gilbert, A. Sommers, A. Thompson, L. Cummins, S. Mittal, S. Rahimi, M. Seale, J. Jaboure, T. Arnold, and J. Church, "AAD-LLM: Adaptive Anomaly Detection Using Large Language Models," Nov. 2024.